{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ikhIvrku-i-L"
   },
   "source": [
    "# Deep & Cross Network (DCN)\n",
    "\n",
    "**Learning Objectives**\n",
    "  1. Build a unified model class whose loss is the mean squared error.\n",
    "  2. Split the data into training set and testing set.\n",
    "  3. Train a DCN model with a stacked structure.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q-rOX95bAye4"
   },
   "source": [
    "##Introduction\n",
    "\n",
    "**What are feature crosses and why are they important?** Imagine that we are building a recommender system to sell a blender to customers. Then, a customer's past purchase history such as `purchased_bananas` and `purchased_cooking_books`, or geographic features, are single features. If one has purchased both bananas **and** cooking books, then this customer will more likely click on the recommended blender. The combination of `purchased_bananas` and `purchased_cooking_books` is referred to as a **feature cross**, which provides additional interaction information beyond the individual features.\n",
    "<img src='./assets/blender-draw.gif' width='50%'>\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "**What are the challenges in learning feature crosses?** In Web-scale applications, data are mostly categorical, leading to large and sparse feature space. Identifying effective feature crosses in this setting often requires\n",
    "manual feature engineering or exhaustive search. Traditional feed-forward multilayer perceptron (MLP) models are universal function approximators; however, they cannot efficiently approximate even 2nd or 3rd-order feature crosses [[1](https://arxiv.org/pdf/2008.13535.pdf), [2](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/18fa88ad519f25dc4860567e19ab00beff3f01cb.pdf)].\n",
    "\n",
    "**What is Deep & Cross Network (DCN)?** DCN was designed to learn explicit and bounded-degree cross features more effectively. It starts with an input layer (typically an embedding layer), followed by a *cross network* containing multiple cross layers that models explicit feature interactions, and then combines\n",
    "with a *deep network* that models implicit feature interactions.\n",
    "\n",
    "\n",
    "*   Cross Network. This is the core of DCN. It explicitly applies feature crossing at each layer, and the highest\n",
    "polynomial degree increases with layer depth. The following figure shows the $(i+1)$-th cross layer.\n",
    "\n",
    "<img src='./assets/dcn-formula.png' width='50%'>\n",
    "\n",
    "*   Deep Network. It is a traditional feedforward multilayer perceptron (MLP).\n",
    "\n",
    "The deep network and cross network are then combined to form DCN [[1](https://arxiv.org/pdf/2008.13535.pdf)]. Commonly, we could stack a deep network on top of the cross network (stacked structure); we could also place them in parallel (parallel structure). \n",
    "\n",
    "\n",
    "<img src='./assets/dcn-parallel.png' width='50%'>\n",
    "<img src='./assets/dcn-stack.png' width='50%'>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6OlIGoADAhZg"
   },
   "source": [
    "In the following, we will first show the advantage of DCN with a toy example, and then we will walk you through some common ways to utilize DCN using the MovieLen-1M dataset.\n",
    "\n",
    "This notebook demonstrates how to use Deep & Cross Network (DCN) to effectively learn feature crosses.\n",
    "\n",
    "Each learning objective will correspond to a __#TODO__ in the [student lab notebook](https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/courses/machine_learning/deepdive2/recommendation_systems/labs/dcn.ipynb) -- try to complete that notebook first before reviewing this solution notebook.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Az-y_3qxH3gA"
   },
   "source": [
    "Let's first install and import the necessary packages for this notebook.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:08:59.840174Z",
     "iopub.status.busy": "2020-11-30T12:08:59.837190Z",
     "iopub.status.idle": "2020-11-30T12:09:02.795223Z",
     "shell.execute_reply": "2020-11-30T12:09:02.794642Z"
    },
    "id": "PjfZWVEWAmxS"
   },
   "outputs": [],
   "source": [
    "# Installing the necessary libraries.\n",
    "!pip install -q tensorflow-recommenders\n",
    "!pip install -q --upgrade tensorflow-datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:09:02.801752Z",
     "iopub.status.busy": "2020-11-30T12:09:02.801039Z",
     "iopub.status.idle": "2020-11-30T12:09:10.015458Z",
     "shell.execute_reply": "2020-11-30T12:09:10.015883Z"
    },
    "id": "DqsyLA0UHeCl"
   },
   "outputs": [],
   "source": [
    "# Importing the necessary modules.\n",
    "import pprint\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.axes_grid1 import make_axes_locatable\n",
    "\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import tensorflow_datasets as tfds\n",
    "\n",
    "import tensorflow_recommenders as tfrs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tHCgOeoHBGbb"
   },
   "source": [
    "## Toy Example\n",
    "To illustrate the benefits of DCN, let's work through a simple example. Suppose we have a dataset where we're trying to model the likelihood of a customer clicking on a blender Ad, with its features and label described as follows.\n",
    "\n",
    "| Features / Label        | Description           | Value Type / Range  |\n",
    "| ------------- |-------------| -----|\n",
    "| $x_1$ = country | the country this customer lives in | Int in [0, 199] |\n",
    "| $x_2$ = bananas | # bananas the customer has purchased |Int in   [0, 23] |\n",
    "| $x_3$ = cookbooks | # cooking books the customer has purchased     |Int in  [0, 5] |\n",
    "| $y$ | the likelihood of clicking on a blender Ad     |   -- |\n",
    "\n",
    "Then, we let the data follow the following underlying distribution:\n",
    "$$y = f(x_1, x_2, x_3) = 0.1x_1 + 0.4x_2+0.7x_3 + 0.1x_1x_2+3.1x_2x_3+0.1x_3^2$$\n",
    "\n",
    "where the likelihood $y$ depends linearly both on features $x_i$'s, but also on multiplicative interactions between the $x_i$'s. In our case, we would say that the likelihood of purchasing a blender ($y$) depends not just on buying bananas ($x_2$) or cookbooks ($x_3$), but also on buying bananas and cookbooks *together* ($x_2x_3$)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HO6d-0zoHrz8"
   },
   "source": [
    "We can generate the data for this as follows:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-fi2ya4P_hab"
   },
   "source": [
    "### Synthetic data generation\n",
    "\n",
    "We first define $f(x_1, x_2, x_3)$ as described above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:09:10.023054Z",
     "iopub.status.busy": "2020-11-30T12:09:10.022427Z",
     "iopub.status.idle": "2020-11-30T12:09:10.024158Z",
     "shell.execute_reply": "2020-11-30T12:09:10.024569Z"
    },
    "id": "9rT3f6C3GX0u"
   },
   "outputs": [],
   "source": [
    "def get_mixer_data(data_size=100_000, random_seed=42):\n",
    "  # We need to fix the random seed\n",
    "  # to make notebook runs repeatable.\n",
    "  rng = np.random.RandomState(random_seed)\n",
    "  country = rng.randint(200, size=[data_size, 1]) / 200.\n",
    "  bananas = rng.randint(24, size=[data_size, 1]) / 24.\n",
    "  coockbooks = rng.randint(6, size=[data_size, 1]) / 6.\n",
    "\n",
    "  x = np.concatenate([country, bananas, coockbooks], axis=1)\n",
    "\n",
    "  # # Create 1st-order terms.\n",
    "  y = 0.1 * country + 0.4 * bananas + 0.7 * coockbooks\n",
    "\n",
    "  # Create 2nd-order cross terms.\n",
    "  y += 0.1 * country * bananas + 3.1 * bananas * coockbooks + (\n",
    "        0.1 * coockbooks * coockbooks)\n",
    "\n",
    "  return x, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JXtUSs9E3uRG"
   },
   "source": [
    "Let's generate the data that follows the distribution, and split the data into 90% for training and 10% for testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:09:10.029207Z",
     "iopub.status.busy": "2020-11-30T12:09:10.028490Z",
     "iopub.status.idle": "2020-11-30T12:09:10.039870Z",
     "shell.execute_reply": "2020-11-30T12:09:10.040270Z"
    },
    "id": "vrQWVYajgmNV"
   },
   "outputs": [],
   "source": [
    "# Generating the data.\n",
    "x, y = get_mixer_data()\n",
    "num_train = 90000\n",
    "train_x = x[:num_train]\n",
    "train_y = y[:num_train]\n",
    "eval_x = x[num_train:]\n",
    "eval_y = y[num_train:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MszQC-KJLhVK"
   },
   "source": [
    "### Model construction\n",
    "\n",
    "We're going to try out both cross network and deep network to illustrate the advantage a cross network can bring to recommenders. As the data we just created only contains 2nd-order feature interactions, it would be sufficient to illustrate with a single-layered cross network. If we wanted to model higher-order feature interactions, we could stack multiple cross layers and use a multi-layered cross network. The two models we will be building are:\n",
    "1.   Cross Network with only one cross layer;\n",
    "2.   Deep Network with wider and deeper ReLU layers. \n",
    "\n",
    "We first build a unified model class whose loss is the mean squared error. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:09:10.046747Z",
     "iopub.status.busy": "2020-11-30T12:09:10.046096Z",
     "iopub.status.idle": "2020-11-30T12:09:10.047928Z",
     "shell.execute_reply": "2020-11-30T12:09:10.048341Z"
    },
    "id": "bwgAH2FTR4Fe"
   },
   "outputs": [],
   "source": [
    "class Model(tfrs.Model):\n",
    "\n",
    "  # TODO 1: Here is your code.\n",
    "  def __init__(self, model):\n",
    "    super().__init__()\n",
    "    self._model = model\n",
    "    self._logit_layer = tf.keras.layers.Dense(1)\n",
    "\n",
    "    self.task = tfrs.tasks.Ranking(\n",
    "      loss=tf.keras.losses.MeanSquaredError(),\n",
    "      metrics=[\n",
    "        tf.keras.metrics.RootMeanSquaredError(\"RMSE\")\n",
    "      ]\n",
    "    )\n",
    "\n",
    "  def call(self, x):\n",
    "    x = self._model(x)\n",
    "    return self._logit_layer(x)\n",
    "\n",
    "  def compute_loss(self, features, training=False):\n",
    "    x, labels = features\n",
    "    scores = self(x)\n",
    "\n",
    "    return self.task(\n",
    "        labels=labels,\n",
    "        predictions=scores,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QAKBq2QxOdM0"
   },
   "source": [
    "Then, we specify the cross network (with 1 cross layer of size 3) and the ReLU-based DNN (with layer sizes [512, 256, 128]):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:09:10.054582Z",
     "iopub.status.busy": "2020-11-30T12:09:10.052628Z",
     "iopub.status.idle": "2020-11-30T12:09:10.257267Z",
     "shell.execute_reply": "2020-11-30T12:09:10.256540Z"
    },
    "id": "EwBwSHz_N3pW"
   },
   "outputs": [],
   "source": [
    "# Specifying the cross network and ReLU based DNN.\n",
    "crossnet = Model(tfrs.layers.dcn.Cross())\n",
    "deepnet = Model(\n",
    "    tf.keras.Sequential([\n",
    "      tf.keras.layers.Dense(512, activation=\"relu\"),\n",
    "      tf.keras.layers.Dense(256, activation=\"relu\"),\n",
    "      tf.keras.layers.Dense(128, activation=\"relu\")\n",
    "    ])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2EtaI5lh4X0B"
   },
   "source": [
    "### Model training\n",
    "Now that we have the data and models ready, we are going to train the models. We first shuffle and batch the data to prepare for model training.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:09:10.268322Z",
     "iopub.status.busy": "2020-11-30T12:09:10.267630Z",
     "iopub.status.idle": "2020-11-30T12:09:10.270803Z",
     "shell.execute_reply": "2020-11-30T12:09:10.271181Z"
    },
    "id": "X6gD-NTF4eoj"
   },
   "outputs": [],
   "source": [
    "# Preparing large set of elements for model training.\n",
    "train_data = tf.data.Dataset.from_tensor_slices((train_x, train_y)).batch(1000)\n",
    "eval_data = tf.data.Dataset.from_tensor_slices((eval_x, eval_y)).batch(1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JYm5bmmgPVZu"
   },
   "source": [
    "Then, we define the number of epochs as well as the learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:09:10.275068Z",
     "iopub.status.busy": "2020-11-30T12:09:10.274419Z",
     "iopub.status.idle": "2020-11-30T12:09:10.276017Z",
     "shell.execute_reply": "2020-11-30T12:09:10.276413Z"
    },
    "id": "nFhrC7fV6szW"
   },
   "outputs": [],
   "source": [
    "epochs = 100\n",
    "learning_rate = 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zbRiVPJtPz-1"
   },
   "source": [
    "Alright, everything is ready now and let's compile and train the models. You could set `verbose=True` if you want to see how the model progresses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:09:10.285983Z",
     "iopub.status.busy": "2020-11-30T12:09:10.285329Z",
     "iopub.status.idle": "2020-11-30T12:09:23.376885Z",
     "shell.execute_reply": "2020-11-30T12:09:23.377555Z"
    },
    "id": "F8ZXXbmKuB8p"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer model is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because its dtype defaults to floatx.\n",
      "\n",
      "If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n",
      "\n",
      "To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer ranking is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because its dtype defaults to floatx.\n",
      "\n",
      "If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n",
      "\n",
      "To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fb9dc772da0>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compiling and training the model.\n",
    "crossnet.compile(optimizer=tf.keras.optimizers.Adagrad(learning_rate))\n",
    "crossnet.fit(train_data, epochs=epochs, verbose=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:09:23.388247Z",
     "iopub.status.busy": "2020-11-30T12:09:23.387086Z",
     "iopub.status.idle": "2020-11-30T12:10:09.136226Z",
     "shell.execute_reply": "2020-11-30T12:10:09.136616Z"
    },
    "id": "Tzg3KLKW2sdA"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer model_1 is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because its dtype defaults to floatx.\n",
      "\n",
      "If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n",
      "\n",
      "To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer ranking_1 is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because its dtype defaults to floatx.\n",
      "\n",
      "If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n",
      "\n",
      "To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fb9dc4a1860>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compiling and training the model.\n",
    "deepnet.compile(optimizer=tf.keras.optimizers.Adagrad(learning_rate))\n",
    "deepnet.fit(train_data, epochs=epochs, verbose=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xxWfaY6H7Bmp"
   },
   "source": [
    "### Model evaluation\n",
    "We verify the model performance on the evaluation dataset and report the Root Mean Squared Error (RMSE, the lower the better)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:10:09.142320Z",
     "iopub.status.busy": "2020-11-30T12:10:09.141654Z",
     "iopub.status.idle": "2020-11-30T12:10:09.419311Z",
     "shell.execute_reply": "2020-11-30T12:10:09.419725Z"
    },
    "id": "l4PM-goX6FoD"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CrossNet(1 layer) RMSE is 0.0003 using 18 parameters.\n",
      "DeepNet(large) RMSE is 0.0350 using 166403 parameters.\n"
     ]
    }
   ],
   "source": [
    "crossnet_result = crossnet.evaluate(eval_data, return_dict=True, verbose=False)\n",
    "print(f\"CrossNet(1 layer) RMSE is {crossnet_result['RMSE']:.4f} \"\n",
    "      f\"using {crossnet.count_params()} parameters.\")\n",
    "\n",
    "deepnet_result = deepnet.evaluate(eval_data, return_dict=True, verbose=False)\n",
    "print(f\"DeepNet(large) RMSE is {deepnet_result['RMSE']:.4f} \"\n",
    "      f\"using {deepnet.count_params()} parameters.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6_Ig-Gnm7-JD"
   },
   "source": [
    "We see that the cross network achieved **magnitudes lower RMSE** than a ReLU-based DNN, with **magnitudes fewer parameters**. This has suggested the efficieny of a cross network in learning feaure crosses.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XsCsY1-Us-_e"
   },
   "source": [
    "### Model understanding\n",
    "We already know what feature crosses are important in our data, it would be fun to check whether our model has indeed learned the important feature cross. This can be done by visualizing the learned weight matrix in DCN. The weight $W_{ij}$ represents the learned importance of interaction between feature $x_i$ and $x_j$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:10:09.427183Z",
     "iopub.status.busy": "2020-11-30T12:10:09.426561Z",
     "iopub.status.idle": "2020-11-30T12:10:09.667195Z",
     "shell.execute_reply": "2020-11-30T12:10:09.667699Z"
    },
    "id": "N8dga2Qck5IV"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/ipykernel_launcher.py:11: UserWarning: FixedFormatter should only be used together with FixedLocator\n",
      "  # This is added back by InteractiveShellApp.init_path()\n",
      "/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/ipykernel_launcher.py:12: UserWarning: FixedFormatter should only be used together with FixedLocator\n",
      "  if sys.path[0] == '':\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 648x648 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAE/CAYAAACuHMMLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAqfklEQVR4nO3deZxcVZ3+8c+TsC8iEAQGZFFRFBgRAogwsrgAiqCDDiDuCuoIiLigDgPKbCiOG6gQmAzigigazSjrT1SQTRJl38UAiQoJQUSEYJLn98c5DUXT3VVJdXdVdT1vX/Xqqntv3XvqRL516nvOPUe2iYiI/jKp0wWIiIjxl+AfEdGHEvwjIvpQgn9ERB9K8I+I6EMJ/hERfSjBPyKiDyX4R0T0oQT/iIg+lOAfEdEiSRrqeS9K8I+IaIEk2bakPSVNdY/PjZPgHxHRghr4Xwd8GViv0+Vpl3r8yysiYlxIehYwE3i77dskbQ1saPviDhdtuaTlHxExDEmT69/1gceBh4E9JJ0BHA98Q9LbOljE5ZbgHxExiKR1JK1ie4mk5wHn2v4T8APgRfX1m4APATtJWqGDxV0uPVfgiIixJGk14CPlqf4V+BOwEMD21xo6fncDjgU+ZHtxxwq8nNLyjwhgYg1jbNMi4EpgFeCjwEbAHQM7a+B/HiXtc4ztizpSyjalwzcinjKMEVhg+/pOl6kTJE2yvbTm+vcE9gE2A14C/BcwBbgbuB+43vZ9A3XXqTIvr7T8I2KgNfta4FRg/U6XpxNqEF8q6e+AdeoonpmUtM9DwDMoMXNzYAXb90Gpuw4VuS1p+UfEwDDG84D32p4t6cXAOsB1thd2tnTjp47j/3dgASXoHwtsSvkV8BDwOduL6rE92eIfkJZ/RAAYuAnYXdJ04ATgi8C+nSzUeJL0HOCTwDtsvwKYBxwNXAFcSMn9bzJwfC8Hfkjwj+hLAx26kjaTtKrt+cCPgWcB59jeH/gCsJukyX3SAfwIcB/wIIDtI4FnU74ALgGOs33H8G/vLRnqGdGHao7/NZQUxwWS1gSOtf09AEm7AB8GPmx7SQeLOmYaOrlXpMTC+ZRUz7aSHrL9IPA/wHNqK39B50o7+pLzj+hDknYCTgf+EXgLcCBwO/BeYClwBjDN9o87VshxIGk/4K2UuXo+C6wLvBG4mRLsDwOOsn1+xwo5RhL8I/pITd8I+AfgAWAD4DPAe4CPA6tRvgwm217Y652aI5G0HaVl/35gY8oX3ynAPcCOwPOA821f0rFCjqGkfSL6QEMQX9X2X4Ff1O2HAZ+w/RtJc4Atgefa/jX0fqdmExsBN9m+CkDSfcC3gP1sn9bRko2DBP+IPlBz2/sCH5F0CXCt7ZnA6pSJygTsQRnpcnMnyzqO5gBIej4wx/Zlkr5DSf30/FDOZpL2iegDkrYEPgVcTLlZaVvg28D1wOeANYGzbJ/boSKOGUnPBFa3Pa92cr8MWImS4/9P4M/AbEqO/zTgQNvXdKi44ybBP2ICqy365wPXACfY/ly9g3V3ytQF37H9E0kr21400Vq7klYFPg/cCtxGuXfhU5R+jXsoHdu7AVtQhnV+zfZ5nSjreEvwj+gD9cat1wKb2n6szk+/D7AXcLTtP3S0gGNI0j6U0UyLgTtsf6Zu/wKwie0D6uu1bD/UuZKOrwT/iAmmYfz684G1bV9dt0+jtPi3t/2wpA2ASbZ/38HijhlJK9heXFv/W1NGMy0FPjlws5aknwHvqytzTahfPc3kDt+ICaZhrdnvA0dJ+omk59k+jDJNwe2S1rD9x4kY+CWtK2nNGvifA/y05vC/CDwG7C1pB0nbUKZrWAQTfmTT02S0T8QEI2lH4DjgVZTx/KcD/y7pONtH1DtapwI/71wpx0Zt5R8FrFYXYnmAJxdiuUzSSpR7Gt4B/I5yB/OcjhS2w9Lyj5h45lBuXNqashjJNpTRLd+RtLXt99n++QSdr+cx4HLKRHUfo0y/fOvATts/pdzIdSdlOosfTtB6aCot/4ge15DjX5fSj3c/cL+k44EZtu+t49c/ROn0BCZemqNhIZYLKV8Cr6WM7JkqaS5l/P7vKEM63z8wVfVEq4dWJfhH9Lga+PcDjgHWlHRsvYHrOkrOfzJlauaP2L51pHP1qoaFWJ4NLK6/bBYBbwf+CKxIma3zmcAf+mmNguEk+Ef0OElbAUdQ5qZ5ASW/P5kyhcPKwH6UMf6/7Fwpx1bDHcwnAvMkLaWkvL5HCfprAifZfhgm/t27rUjwj+hhtaX7QeBvtm8EbpS0hNLhu4LtcyR9r7aKJ2zAk7QZ8AngbbZ/LekzlJz/UZQU0EHAhsDD0L+pnkbp8I3oMYM6KO8HrgaWSnprvVP3h5RpC06QtCGl83OiB7w/U+riEQDbxwBrAMfYvhz4lO3bO1i+rpOWf0QPaejc3RVYC3jA9v/UNMeOlC+Bc22fK+nS2vk74TTUw8qUXzgLJS2kLMQyv+b0v0EZ8YTtBzpZ3m6U4B/RQxpu4DqBEtxeU4P9qTXdsycwWdI3mGArTzVq6OR+F7COpC9TbmB7C2V0z0JKZ+/RHSxmV0vwj+ghNbd9BPA6SqBfG3i9yjq8X5C0AjC7pngmbJpH0ksowzjfB6xDWXLydEref2AhlifuZ5jgKa/lkuAf0cUG8vsNwesvwJGUlaeOBvanzEr5b5JWtP3ZjhR0/K0P3Gb7VwCSFgA/AP7J9v82HpjAP7R0+EZ0MVeStpK0ObBKHau/EXCO7bspI1jOZQJO1zBg4EtQ0naSVgHuAxZL2rJ2cs8Cvk4Zxx8tSPCP6EKSNqnDFZH0Cko++zjgYkl7UKYnOEzSx4GvAD8aaAVPRPULcG/KuP0XUW5ge5CSAjuwTtt8SN0WLciUzhFdSNIalNWlfkmZnOz/6sRkB1BWm9oBeA5lEZLf2r64Y4UdB5I2BWYCh9u+rG4TJfW1EaUuTrN9fudK2VsS/CO6jKTJtpdIWhP4EbABcDBwQ71Z6yPAs2x/rKMFHUe1o/tE2wfV16vafrT2c/ytTuH8cGdL2VuS9onoMjXwT67B7HWU/Pb7Gw5ZCEzpSOE65z5gS0lHANTA/yrgvyVNot7cFa1Lyz+iSzX8AlgDuIhyF+v5lKkKTrT9o44WcJw0zNa5C3AscAdl3qJjKXfu9kU9jLYE/4gOk7QOZbnF3w6xb+ALYHXgYsqv9XfavmWijV8fqR7qflFy+8cC9wJX2j5/otXDeEnwj+igOj3B54CHgLOGmn9m0BfANravGu9yjrVW6mGY9w2+DyJalJx/RIfUFusiStDbENhf0iaDj2voA3jE9lUTbeWpVuuh4fjJA88H7oMYh2JOOAn+EZ23EfBsyhQFR0raYrgDa/5/p9rJOdE0rYfGfhBJL52g9TAuUnERHVJvXPp74AzKlA0HU1q+/1Tn6QeeEvCeCfwUWGR7aSfKPBZSD52RuX0iOmsdYF6dsuFWSQ8C3wI2kHQKcHtDwDuXshTjbzpX3DGTehhnaflHjKOGOWpWrZtuBu6RtJek1WxfSwluzwceq63iZwAzgE8P3N3a61IPnZfRPhHjTGWt2dcAjwJnA6+lLMzyJ8qcNccAH60rUFEXbvmb7as7UuAxknrorAT/iHFUA9hXgDdQWrGXUuag3xuYSpmH/kzbP+5YIcdB6qHzEvwjxkFDZ+XRwBzKKlufBQ62/TtJq9t+pOHvhLxxKfXQPdLhGz2hV4NAQ7nXoszJcxdwKLAe8Cbbd0s6BNha0icpKZAJd9NS6qH7pMM3upqkrXs98Et6DfAjSWsBdwMrUdbfXSJpe0pu+5f1fqUJN3Qx9dCdkvaJrqWyYtPJwAW2v9+LXwIqC7GcQllP9hd1266UseybAisDJ9ue2Yufr1Wph+6T4B9dS9KKwMeAybZP6HR5llUdzvjPlOmIrwJeWV9/Dfg/YBGwuu0/TuSAl3roTkn7RNepqZ5tbP8NOBP4R5Ul/Lpe47w7NYjdB5xAWV92beCrwFuB9Ww/bPuPDcdOGKmH7pcO3+gqknYG3gS8XtKJlJt//ou6eInq3O4dLOKwGnLbewAvo4xVn0FZjvHR2rLdmNLq7crPMBpSD70hLf/ouIa7Pbem5IX/g7JgyUrAhyhDAf9F0gbdGvjhiTlq9gU+T1lI/HDgS8AKNeAdBJwH/Jft2zpY1DGVeugNCf7RUQ2txJ2A7wL/bfsB27+yfQrwZuA/Ka3Gt6jqZJmHI2kD4ADg9cA9lFkqFwEflrQhZXjjUbZndOtnGA2ph96QDt/oCJWpeFetN/JsQQkSvwVm296/HrOS7cfr832BV9j+UMcKPchA4BrIU0tagTIl8eqUvooDgM0p+e2rgPfWfowJJfXQm9Lyj07ZEThN0lsoIz5WAv4e2E7SlwBsP14DCZRZH/eStFY3tBYlrVjHo1vSjpJ2p3Re/g5YBbjD9t3AY8CvgJMmYsBLPfSutPyjYyR9l5IaOND2jLptbUrr8FLbhzYcuzPwkO2bO1HWRiprzZ5NGaO+MfBDyq+WWygLrF9CWWT8V8BLKS3dn3SksGMo9dDbMtonxtWgcdzfAh4BjpY0y/a9th+s+f+rawfwzbaX2r6yY4UexPZCSTcAPwHuBPahLCj+HsrMlA8AWwB7AifantWpso6l1ENvS8s/xl1txW8M/ML2/XVI557AHsAOlHzxN7txzHdDB/VkyqpT/wbsYfsaSRsB/whsC/xgIrdyUw+9Ly3/GFeSdgNOBeZSluk7w/bHJX2WkvvfADi2ywP/y4AlwJeBvwO+JumNtudImkH57+p3nSzrWEo9TAxp+ce4kfRi4DPAB23fJunDlLTADNsXSnohsMT27d16m7+kvShfXm8BrrS9VNJxwF7A223fKWll24s6WtAxlnrofRntE2Nq0MicbYCdgd3q688DtwFvlrS/7Vts3w7deZu/pDWBTwKHuqwuNVDGf6d0bn5XZVnCCT2aJfUwMSTtE2OqpgdeATxi+5s1KPyjpPtt/xD4gqSPUkaJdK3aT7EAuBW4vW5emTKEcUNKzvtM2492poTjI/UwcaTlH+Nhc+AKSTvZPh34DvAuSW8CsH2S7Rs7WsIR1ID3ReAZlD6JjwHYfqyOTPpvYA3bXf0F1q7Uw8SSln+MGZVFO/5i+wxJS4GLJO1t++sq0zW/R9KlwPxunbNH0gsoE5DNsD1b0luBKyWdTfm1si/wadsLO1nOsZZ6mHgS/GNMNASL70u63Pb0mv6/WNIr6hfC+bbv62xJh1e/oCYDqwK7SHqR7ZslTQUOARYDR9i+rFs7qEdD6mFiymifGDWN/+HXTsFPU1KL3wVm1ekaLqZM7bCx7Yc7V9qRSXouZRWxd1HmqPkg8HvKuPXbR3rvRJJ6mLiS849R0TD2e2dJ+wNTbR8NPAT8E7C7pJcD1wOv7ObAXz1EKeupwMPANOBZlJFJz+1kwcZZ6mGCSvCPUVED/96U4LAL8B/1Bq7jKTd0HURZxelntq+Bpw0D7QqSngdgewFwEvAb4H8ord2zqIvKTHSph4kvaZ8YFSpTNJ8NnGP7B3XblZRg/0lJKwEb2L6n2/LCDb9aJlFaud+z/em6bz1K2mMtyg1Ni2z/pXOlHTuph/6Sln8st4GWe53G9wDgfuCvDYe8C9hA0gq2H7d9D3TfDVw14O0KvIqy4tQ7JH2g7psPzAIeBTabyAEv9dBfMtonllsNFq+jLMz9PsqNP6dK2sX2PMoKTpsDqwF/7lxJhzZojprTgV9TZqX8A3CspDWAPwIHAu/s5nsR2pF66E8J/rHcalB4F/AB21dTpmGeAlwo6ULgNcBHbHdd4Icnvrx2pKwZ/E7bV9Vc91xgJ2ATYCvgPydywEs99KcE/2iHgfWANeCJFuSnJc2hrLn77XpDUFfl+AdZC3g5ZUrpq4C7Ka3ezYGPwBPBsZs/w2hIPfSZ5Pxjudl+BDgHeJmkFw4M9aSM7Jlve3Y9rmuDhe2LKXPPv0vSwS5LDP6JEgjXaziuaz/DaEg99J+M9om2qCzc8V7KQiy/pIzpP9I9toBH7bv4FnARsJSymMzMzpZq/KUe+keCf7RN0uqUFbjWB+bU/H/PkbQfpfP6W7ZPGhjN1G+t3dRDf0jOP9pW0z8/73Q52mV7pqTHgOmSfjtwv0K/ST30h7T8IwaR9Crgt7bv6nRZOin1MLEl+EdE9KGM9omI6EMJ/hERfSjBPzpG0mGdLkM3Sr08Xepk9CX4RyflP+ihpV6eLnUyyhL8IyL6UEb79JkpU6Z4k00363QxAFgwfz5T1luv+YHjYGkX/XfwwIIFrDulO9ZKmdwl6+3MXzCf9aZ0x/9X7r57DgsWLHiiYiY/Y1N78aNN3+dH519oe+8xLdwyyE1efWaTTTfj0it+1elidJ1Ff1va6SJ0pdVXSYgYbJedpj7ltRc/xspbHtT0fY/95uTu+Eav8i8bEdEOAV3yC2lZJPhHRLRLvdd9muAfEdGutPwjIvqN0vKPiOg7AiZN7nQpllmCf0REW5S0T0REX0raJyKiD6XlHxHRZ6Tk/CMi+lLSPhER/SZDPSMi+tOk9nP+kqYD+wL32956iP0fBQ6pL1cAXgisZ3uhpDnAw8ASYLHtqYPf/7Qit13iiIh+JkrLv9mjuTOBYWf9tH2S7W1tbwt8AviF7YUNh+xR9zcN/JCWf0REm0anw9f2pZI2a/Hwg4Gz27leWv4REe2Smj9G7VJajfIL4fsNmw1cJGl2q0tepuUfEdGu1tI6UyTNang9zfa05bja64DLB6V8drU9T9KzgIsl3Wr70pFOkuAfEdGO1lv2C1rNxzdxEINSPrbn1b/3S5oB7AiMGPyT9omIaNekyc0fo0DSWsBuwI8atq0uac2B58CrgRubnSst/4iItozOOH9JZwO7U9JDc4HjgRUBbJ9aD3sDcJHtRxreuj4wQ+XXxwrAt21f0Ox6Cf4REe0ahQ5d2we3cMyZlCGhjdvuAl68rNdL8I+IaMfAOP8ek+AfEdGWTO8QEdGfMqtnREQfynz+ERF9Rkn7RET0p7T8IyL6i4BJk9Lyj4joL6qPHpPgHxHRFqEeTPv03m+VPiLpqDp9a0R0MUlNH90mwb+7HQUMGfwl9d7A4ogJKsG/D0l6m6TrJV0n6RuSNpN0Sd32U0mb1OPOlPTGhvf9pf7dXdLPJZ0r6VZJ31JxJPB3wM8k/WzgPZL+W9J1wL9I+mHD+V5Vp3KNiPEk0CQ1fXSb5PzbIGkr4FjgZbYXSFoH+Drwddtfl/Qu4MvA65uc6iXAVsDvgcuBXWx/WdLRlHU5F9TjVgeutv1hlabELZLWsz0feCcwfbQ/Y0SMTMn596U9ge8NBOe6ss7OwLfr/m8Au7Zwnl/Znmt7KXAtsNkwxy2hLt1m2/X8b5H0zHrd84d6k6TDJM2SNGvB/PktFCcilkUvpn3S8h8/i6lftpImASs17FvU8HwJw/+7PGZ7ScPr/wX+D3iM8iW0eKg31aXipgFst/1UL1fpI2JY3Rjcm0nLvz2XAG+StC5ATftcQVlmDeAQ4LL6fA6wfX2+H3WRhiYeBtYcbqft31NSRcdSvggiYrwl599/bN8k6T+AX0haAvwGOAL4X0kfBQZy8QCnAz+qnbUXAI8Mdc5BpgEXSPq97T2GOeZbwHq2b2nns0TE8uvFln+Cf5tsf53SydtozyGOuw94acOmY+r2nwM/bzju8IbnJwMnN7xeY4gi7Er5YomIDujVDt8E/x4maTblF8SHO12WiH7Wi8E/Of8eZnt72y+3vaj50RExZtTCo9kppOmS7pd04zD7d5f0kKRr6+O4hn17S7pN0p2SPt5KkdPyj4hoh0ZtVs8zgVOAs0Y45jLb+z7l8uVu/68ArwLmAtdImmn75pEulpZ/RESbRmOcv+1LgYXLcfkdgTtt32X7ceA7wP7N3pTgHxHRhoEO3xaC/5SBmy3r47DluNzOdSqZ8+sMAwAbAfc2HDO3bhtR0j4REe1qrb93ge2pbVzl18Cmtv8i6TXAD4EtlvdkaflHRLSj5vybPdpl+8+2/1KfnwesKGkKMA94dsOhG9dtI0rLPyKiTeMx1FPSBsB9ti1pR0rj/QHgT8AWkjanBP2DgDc3O1+Cf0REu0Yh9ks6G9id0jcwFzieOg2M7VOBNwLvl7QYeBQ4qE7wuFjS4cCFwGRguu2bml0vwT8iok2j0fK3fXCT/adQhoIOte884LxluV6Cf0REGySN1jj/cZXgHxHRpl6c3iHBPyKiXb0X+xP8IyLalZZ/RES/UYJ/RETfEWJSF67U1UyCf0REm3qw4Z/gHxHRrqR9IiL6jdLyj4joOwImT+696J/gHxHRpqR9IiL6TdI+ERH9R6TlHxHRh1pbo7fbJPhHRLQpN3lFRPSb5PwjIvpPcv4REX2qB2M/vbf8TEREl5k0SU0fzUiaLul+STcOs/8QSddLukHSFZJe3LBvTt1+raRZrZQ5Lf8+s9Tmr48v6XQxus4zVl2x00XoSmvvcHini9B1Ft12z1M3jN6UzmdS1ug9a5j9vwN2s/2gpH2AacBODfv3sL2g1Ysl+EdEtKHk/Ns/j+1LJW02wv4rGl5eBWzczvWS9omIaEsZ59/sMcreDZzf8NrARZJmSzqslROk5R8R0aYWY/uUQfn4abanLfu1tAcl+O/asHlX2/MkPQu4WNKtti8d6TwJ/hER7VDLN3ktsD21rUtJfw+cAexj+4GB7bbn1b/3S5oB7AiMGPyT9omIaMPAOP+xTvtI2gT4AfBW27c3bF9d0poDz4FXA0OOGGqUln9ERJtGKbifDexOSQ/NBY4HVgSwfSpwHLAu8NV6vcX1l8T6wIy6bQXg27YvaHa9BP+IiDaN0mifg5vsfw/wniG23wW8+OnvGFmCf0REO1rP+XeVBP+IiDYoUzpHRPSnHoz9Cf4REe2a1IPRP8E/IqJNPRj7E/wjItohweR0+EZE9J90+EZE9KEejP0J/hER7RBluGevSfCPiGiHlJx/REQ/StonIqLPiIzzj4joSz0Y+xP8IyLalaGeERF9Jjd5RUT0qd4L/Qn+ERFtS9onIqLPlNE+nS7Fskvwj4hoh9STK3lN6nQBIiJ6naSmjxbOMV3S/ZJuHGa/JH1Z0p2Srpe0XcO+t0u6oz7e3kqZE/wjItowkPZp9mjBmcDeI+zfB9iiPg4DvgYgaR3geGAnYEfgeElrN7tYgn9ERJtGo+Vv+1Jg4QiH7A+c5eIq4JmSNgT2Ai62vdD2g8DFjPwlAiTnHxHRthYz/lMkzWp4Pc32tGW4zEbAvQ2v59Ztw20fUYJ/REQbluEmrwW2p451eVqVtE9ERJtGI+3TgnnAsxteb1y3Dbd9RF0d/CVtNlzP9xhd70xJbxxh/xxJU8arPBHRG6Tmj1EwE3hbHfXzUuAh238ALgReLWnt2tH76rptRF2R9pG0gu3FnS5HRMSyEhqVKZ0lnQ3sTukbmEsZwbMigO1TgfOA1wB3An8F3ln3LZT0b8A19VQn2B6p4xgYxeAvaTPgAmA2sB1wE/A24GZgqu0FkqYCn7O9u6RPAc8FngPcI+ko4NT6GuD9wO+ByZJOB15G+Smzv+1HJR1KGe60EqUy3mr7r5LeRKm0JZRvxpdLmgycSKnYlYGv2D5N5bfYycCrKB0mj7fwUT8maR/gUeDNtu+U9Drg2FqWB4BDbN9XP+Mm9TNtAnzR9pdrff2Q8lNtFeBLAx0/kv4CfAnYt15j/3qu4a6xWz0ewMDLbT/cwueIiNEgRuUmL9sHN9lv4APD7JsOTF+W64122ucFwFdtvxD4M/DPTY5/EfDK+qG/DPzC9ot58ssDypjWr9jeCvgTcEDd/gPbO9TjbwHeXbcfB+xVt+9Xt72b8kWwA7ADcKikzYE31DK/iPJF9bIWPuNDtrcBTgG+WLf9Enip7ZcA3wE+1nD8lpShWAPjb1es299le3tgKnCkpHXr9tWBq2r5LwUObXKNjwAfsL0t8A+UL4ynkHSYpFmSZj2wYEELHzEilsWkFh7dZrTLdK/ty+vzbwK7Njl+pu2BYLUn9aYF20tsP1S3/872tfX5bGCz+nxrSZdJugE4BNiqbr8cOLP+Mphct72akiu7FrgaWJfypfJy4Ox6vd8Dl7TwGc9u+Ltzfb4xcGEty0cbygLwE9uLbC8A7gfWr9uPlHQdcBXlF8AWdfvjwI+H+LzDXeNy4POSjgSeOVT6zPY021NtT113SrosIkaTGLcO31E12sHfQ7xe3HCdVQbtf6SFcy5qeL6EJ1NVZwKH11b4pwfObft9lPTIs4HZtUUt4Ajb29bH5rYvau0jPY2HeH4ycEoty3t56ud8Wvkl7Q68Eti5tvB/0/Cev9Wfd4M/75DXsH0i8B5gVeBySVsu5+eKiOU0Snf4jqvRDv6bSBpoDb+ZkqqYA2xftx0w1Juqn1Ly/EiaLGmtJtdaE/hDTaMcMrBR0nNtX237OGA+5UvgQuD9AykXSc+XtDolrXJgvd6GwB4tfMYDG/5eWZ+vxZNDq1qZV2Mt4MHaR7El8NIW3/O0a9TPe4Ptz1A6fBL8I8bRwDj/Zo9uM9rB/zbgA5JuAdampHE+DXyp3tm2ZIT3fhDYo6Y1ZlPy8CP5V0oK53Lg1obtJ0m6oQ4RvQK4DjiD0vH867r9NEqLegZwR913Fk8G85GsLen6Wt4P1W2fAr4naTbQSlL9AsovgFsoHdFXtfCe4a5xlKQba5n+BpzfwrkiYhT1YstfT2YY2jxRGe3zY9tbj8oJY0xsu932vugXrXzX9JdnrLpi84P60No7HN7pInSdRbd9l6V/vf+JcL7BFlv7bV/8ftP3nbTvlrO76Q7frhjnHxHRq8qsnl3YtG9i1IK/7TnAhGj1S5oBbD5o8zG2m941FxH9pxuHcjaTlv8QbL+h02WIiN4gdWeHbjMJ/hERberBrE+Cf0REu3qw4Z/gHxHRjr7v8I2I6EuCyT3Y45vgHxHRJrW6kGMXSfCPiGhDSft0uhTLLsE/IqJNCf4REX2oG6dsbibBPyKiDerRDt8eLHJERHeZJDV9tELS3pJuk3SnpI8Psf8Lkq6tj9sl/alh35KGfTObXSst/4iINoxWh29da/wrlDXF5wLXSJpp++aBY2x/qOH4I4CXNJzi0bqca0vS8o+IaJPU/NGCHYE7bd9l+3HKWt37j3D8wTy5rOwyS/CPiGiDEJPV/NGCjYB7G17Prduefk1pU8rMw43rjq8iaZakqyS9vtnFkvaJiGhH6yt1TakrGg6YZnvacl71IOBc242rI25qe56k5wCXSLrB9m+HO0GCf0REm1rs0F3QZCWveZQ1xwdszJPrdg92EPCBxg2259W/d0n6OaU/YNjgn7RPREQbxKjl/K8BtpC0uaSVKAH+aaN2JG1JWSP9yoZta0tauT6fAuxCWZt8WGn5R0S0aTRm9bS9WNLhwIXAZGC67ZsknQDMsj3wRXAQ8B0/dQH2FwKnSVpKadSf2DhKaCgJ/hERbRAweZRu8LV9HnDeoG3HDXr9qSHedwWwzbJcK8E/IqIdyvQOERF9qfdCf4J/RERbspJXRESfypTOERF9R8n5R0T0G9GbN0wl+EdEtCkt/+h6kyRWXzn/7IOtd8jXO12ErjT3l1/sdBG6ziv+4cqnbeu90J/gHxHRFolWZ+3sKgn+ERFtStonIqIP9V7oT/CPiGhbDzb8E/wjItpRJnbrveif4B8R0RahHkz8JPhHRLSpBxv+Cf4REe0od/j2XvRP8I+IaEfryzR2lQT/iIg2ZUrniIg+U+bz73Qpll0vTkYXEdFV1ML/WjqPtLek2yTdKenjQ+x/h6T5kq6tj/c07Hu7pDvq4+3NrpWWf0REm0Yj6yNpMvAV4FXAXOAaSTNt3zzo0HNsHz7ovesAxwNTAQOz63sfHO56aflHRLRh4CavZo8W7Ajcafsu248D3wH2b7EYewEX215YA/7FwN4jvSHBPyKiLa0kfQQwRdKshsdhg060EXBvw+u5ddtgB0i6XtK5kp69jO99QtI+ERHtaH2o5wLbU9u82v8BZ9teJOm9wNeBPZfnRGn5R0S0SS08WjAPeHbD643rtifYfsD2ovryDGD7Vt87WIJ/REQbylBPNX204BpgC0mbS1oJOAiY+ZRrSRs2vNwPuKU+vxB4taS1Ja0NvLpuG1bSPhERbRqN0T62F0s6nBK0JwPTbd8k6QRglu2ZwJGS9gMWAwuBd9T3LpT0b5QvEIATbC8c6XoJ/hERbRqtWT1tnwecN2jbcQ3PPwF8Ypj3Tgemt3qtBP+IiDb14OwOCf4REe3qwdif4B8R0Q6RBdwjIvpPpnSOiOhPPRj7E/wjItrWg9E/wT8ioi0t38TVVRL8IyLasAzTN3SVBP+IiHb1YPRP8I+IaNNo3eE7nhL8IyLa1IMp/wT/iIi29Og4/45M6SxpM0k3juP1zpT0xjG+xjsknTJW1x7u/BHReaO1gPt4GtOWv6QVbC8ey2tERHRSmd6h06VYdk1b/rWVfqukb0m6pa4buZqkOZKm1GOmSvp5ff4pSd+QdDnwDUnrS5oh6br6eFk99WRJp0u6SdJFklat7z9U0jX12O9LWq1uf5OkG+v2S+u2yZJOqsdfX5c1Q8Upkm6T9P+AZzX5jDtIuqKe+1eS1pS0iqT/lXSDpN9I2qMeO+T2Qed7raQrB+oHeGVds/N2SfuOdJ5lPf9Q9TLE8YcNrBu6YMH8kaoiIpbDKK3kNa5abfm/AHi37cslTQf+ucnxLwJ2tf2opHOAX9h+g6TJwBrA2sAWwMG2D5X0XeAA4JvAD2yfDiDp34F3AycDxwF72Z4n6Zn1Ou8GHrK9g6SVgcslXQS8pJb5RcD6wM0MM891XTHnHOBA29dIegbwKPBBwLa3kbQlcJGk5wMfGGb7wPneABwNvMb2g3XCp82AHYHnAj+T9LwRzrOs5x+qXp7C9jRgGsB220/1MP9mEbGcenFit1Zz/vfavrw+/yawa5PjZ9p+tD7fE/gagO0lth+q239n+9r6fDYlQAJsLekySTcAhwBb1e2XA2dKOpSyyg2UpcreJula4GpgXcqXysspixwvsf174JIRyvoC4A+2r6ll/HNNVe1aPyu2bwXuBp4/wvaBz3oM8FrbDzZc47u2l9q+A7gL2HIUzz9UvUTEOJKaP7pNq8F/cGvRlGXEBt6/yqD9j7RwzkUNz5fw5K+QM4HDbW8DfHrg3LbfBxxLWaR4tqR1Kb+mjrC9bX1sbvui1j7SmPgtsCZPBusBQ9XfqJx/mHqJiHHUi2mfVoP/JpJ2rs/fDPwSmMOTK8cfMMJ7fwq8H57I0a/V5FprAn+QtCKl5U9973NtX12XNJtPCXYXAu+vxyLp+ZJWBy4FDqzX2xB4Wt68wW3AhpJ2qOdYU9IKwGUD169pl03qscNth9JKPwA4S9JWDdd4k6RJkp4LPKfJeZbp/MPUS0SMp1GK/pL2rn2Vd0r6+BD7j5Z0c+3j/KmkTRv2LZF0bX3MHPzewVoN/rcBH5B0CyVf/zVKq/xLkmZRWu7D+SCwR03jzKbk4Ufyr5QUzuXArQ3bT6qdoDcCVwDXAWdQ8vm/rttPo/yCmAHcUfedBVw53MVsPw4cCJws6TrgYsqvja8Ck2q5zwHeYXvRCNsHzncrJXh/rwZ7gHuAXwHnA++z/dgonn+oeomIcVJie/tDPWuf6FeAfShx8mBJg+Plb4Cptv8eOBf4bMO+RxuyIPs1vZ49cgZC0mbAj21v3bT00fW2236qL7vymk4Xo+ts8NazOl2ErjRn+iHND+ozr/iHnbj217OfiObbbLudf3jR5SO9BYDnrb/abNtTh9tfsyufsr1Xff0JANv/NczxLwFOsb1Lff0X22u0+jk6cpNXRMSEMjppn42Aextez63bhvNuSjZhwCp1SPdVkl7f7GJNh3rangNMiFa/pBnA5oM2H2P7wk6UJyImgpbv4J1S0+QDptVh2Mt+RektwFRgt4bNm9Yh388BLpF0g+3fDneOvprbx/YbOl2GiJh4WhzKuWCktA8wj6cO2Ni4bht0Lb0S+Bdgt0H9gfPq37tUbrp9CWWE4JCS9omIaMPA9A6jMM7/GmALSZvXm08PAp4yaqfm+U8D9rN9f8P2teuNrqjMLLALZcDLsPqq5R8RMRZGY+I224slHU4Zwj4ZmG77JkknALNszwROosyS8L16V/E9dWTPC4HTJC2lNOpPtJ3gHxExlkbrDl7b5wHnDdp2XMPzVw7zviuAbZblWgn+ERFt6sY7eJtJ8I+IaEeXzt3TTIJ/REQbSodv70X/BP+IiDb1XuhP8I+IaFsPNvwT/CMi2tWNa/Q2k+AfEdGmtPwjIvpMt67U1UyCf0REm5L2iYjoR70X+xP8IyLa1YOxP8E/IqI9YlIPJv0T/CMi2jAwpXOvyXz+ERF9KC3/iIg29WLLP8E/IqIdIjn/iIh+IzLaJyKiP/Vg9E/wj4hoU+7wjYjoQz2Y8k/wj4hoV4J/REQf6sW0j2x3ugwxjiTNB+7udDmqKcCCTheiC6Venq6b6mRT2+sNvJB0AaV8zSywvffYFWvZJPhHx0iaZXtqp8vRbVIvT5c6GX2Z3iEiog8l+EdE9KEE/+ikaZ0uQJdKvTxd6mSUJecfEdGH0vKPiOhDCf4REX0owT8iog8l+EdE9KEE/4iIPvT/AQZ4Jo98AEZvAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 288x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "mat = crossnet._model._dense.kernel\n",
    "features = [\"country\", \"purchased_bananas\", \"purchased_cookbooks\"]\n",
    "\n",
    "plt.figure(figsize=(9,9))\n",
    "im = plt.matshow(np.abs(mat.numpy()), cmap=plt.cm.Blues)\n",
    "ax = plt.gca()\n",
    "divider = make_axes_locatable(plt.gca())\n",
    "cax = divider.append_axes(\"right\", size=\"5%\", pad=0.05)\n",
    "plt.colorbar(im, cax=cax)\n",
    "cax.tick_params(labelsize=10) \n",
    "_ = ax.set_xticklabels([''] + features, rotation=45, fontsize=10)\n",
    "_ = ax.set_yticklabels([''] + features, fontsize=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bQHVZTu03qvi"
   },
   "source": [
    "Darker colours represent stronger learned interactions - in this case, it's clear that the model learned that purchasing babanas and cookbooks together is important.\n",
    "\n",
    "If you are interested in trying out more complicated synthetic data, feel free to check out [this paper](https://arxiv.org/pdf/2008.13535.pdf)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0wU4FcpfHCZM"
   },
   "source": [
    "## Movielens 1M example\n",
    "We now examine the effectiveness of DCN on a real-world dataset: Movielens 1M [[3](https://grouplens.org/datasets/movielens)]. Movielens 1M is a popular dataset for recommendation research. It predicts users' movie ratings given user-related features and movie-related features. We use this dataset to domenstrate some common ways to utilize DCN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8Rvlem07wfwH"
   },
   "source": [
    "### Data processing\n",
    "\n",
    "The data processing procedure follows a similar procedure as the [basic ranking notebook](https://www.tensorflow.org/recommenders/examples/basic_ranking)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:10:09.673276Z",
     "iopub.status.busy": "2020-11-30T12:10:09.672655Z",
     "iopub.status.idle": "2020-11-30T12:11:15.692491Z",
     "shell.execute_reply": "2020-11-30T12:11:15.692897Z"
    },
    "id": "7Y_n3EPosR4A"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:The handle \"movie_lens\" for the MovieLens dataset is deprecated. Prefer using \"movielens\" instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mDownloading and preparing dataset movie_lens/100k-ratings/0.1.0 (download: 4.70 MiB, generated: 32.41 MiB, total: 37.10 MiB) to /home/kbuilder/tensorflow_datasets/movie_lens/100k-ratings/0.1.0...\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shuffling and writing examples to /home/kbuilder/tensorflow_datasets/movie_lens/100k-ratings/0.1.0.incompleteBFZ1VZ/movie_lens-train.tfrecord\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mDataset movie_lens downloaded and prepared to /home/kbuilder/tensorflow_datasets/movie_lens/100k-ratings/0.1.0. Subsequent calls will reuse this data.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "ratings = tfds.load(\"movie_lens/100k-ratings\", split=\"train\")\n",
    "ratings = ratings.map(lambda x: {\n",
    "    \"movie_id\": x[\"movie_id\"],\n",
    "    \"user_id\": x[\"user_id\"],\n",
    "    \"user_rating\": x[\"user_rating\"],\n",
    "    \"user_gender\": int(x[\"user_gender\"]),\n",
    "    \"user_zip_code\": x[\"user_zip_code\"],\n",
    "    \"user_occupation_text\": x[\"user_occupation_text\"],\n",
    "    \"bucketized_user_age\": int(x[\"bucketized_user_age\"]),\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2Yb3KxrgSHiF"
   },
   "source": [
    "Next, we randomly split the data into 80% for training and 20% for testing.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:11:15.701072Z",
     "iopub.status.busy": "2020-11-30T12:11:15.700382Z",
     "iopub.status.idle": "2020-11-30T12:11:15.703816Z",
     "shell.execute_reply": "2020-11-30T12:11:15.703384Z"
    },
    "id": "a5-l91jR_zEo"
   },
   "outputs": [],
   "source": [
    "# TODO 2: Here is your code.\n",
    "tf.random.set_seed(42)\n",
    "shuffled = ratings.shuffle(100_000, seed=42, reshuffle_each_iteration=False)\n",
    "\n",
    "train = shuffled.take(80_000)\n",
    "test = shuffled.skip(80_000).take(20_000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MRHGa9mESMVz"
   },
   "source": [
    "Then, we create vocabulary for each feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:11:15.709382Z",
     "iopub.status.busy": "2020-11-30T12:11:15.708534Z",
     "iopub.status.idle": "2020-11-30T12:11:56.709858Z",
     "shell.execute_reply": "2020-11-30T12:11:56.710320Z"
    },
    "id": "l9qhEcHq_VfI"
   },
   "outputs": [],
   "source": [
    "feature_names = [\"movie_id\", \"user_id\", \"user_gender\", \"user_zip_code\",\n",
    "                 \"user_occupation_text\", \"bucketized_user_age\"]\n",
    "\n",
    "vocabularies = {}\n",
    "\n",
    "for feature_name in feature_names:\n",
    "  vocab = ratings.batch(1_000_000).map(lambda x: x[feature_name])\n",
    "  vocabularies[feature_name] = np.unique(np.concatenate(list(vocab)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Eti8kNkPSORk"
   },
   "source": [
    "### Model construction\n",
    "\n",
    "The model architecture we will be building starts with an embedding layer, which is fed into a cross network followed by a deep network. The embedding dimension is set to 32 for all the features. You could also use different embedding sizes for different features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:11:56.723305Z",
     "iopub.status.busy": "2020-11-30T12:11:56.722559Z",
     "iopub.status.idle": "2020-11-30T12:11:56.724755Z",
     "shell.execute_reply": "2020-11-30T12:11:56.724263Z"
    },
    "id": "6lrDcBjiwnHU"
   },
   "outputs": [],
   "source": [
    "class DCN(tfrs.Model):\n",
    "\n",
    "  def __init__(self, use_cross_layer, deep_layer_sizes, projection_dim=None):\n",
    "    super().__init__()\n",
    "\n",
    "    self.embedding_dimension = 32\n",
    "\n",
    "    str_features = [\"movie_id\", \"user_id\", \"user_zip_code\",\n",
    "                    \"user_occupation_text\"]\n",
    "    int_features = [\"user_gender\", \"bucketized_user_age\"]\n",
    "\n",
    "    self._all_features = str_features + int_features\n",
    "    self._embeddings = {}\n",
    "\n",
    "    # Compute embeddings for string features.\n",
    "    for feature_name in str_features:\n",
    "      vocabulary = vocabularies[feature_name]\n",
    "      self._embeddings[feature_name] = tf.keras.Sequential(\n",
    "          [tf.keras.layers.experimental.preprocessing.StringLookup(\n",
    "              vocabulary=vocabulary, mask_token=None),\n",
    "           tf.keras.layers.Embedding(len(vocabulary) + 1,\n",
    "                                     self.embedding_dimension)\n",
    "    ])\n",
    "      \n",
    "    # Compute embeddings for int features.\n",
    "    for feature_name in int_features:\n",
    "      vocabulary = vocabularies[feature_name]\n",
    "      self._embeddings[feature_name] = tf.keras.Sequential(\n",
    "          [tf.keras.layers.experimental.preprocessing.IntegerLookup(\n",
    "              vocabulary=vocabulary, mask_value=None),\n",
    "           tf.keras.layers.Embedding(len(vocabulary) + 1,\n",
    "                                     self.embedding_dimension)\n",
    "    ])\n",
    "\n",
    "    if use_cross_layer:\n",
    "      self._cross_layer = tfrs.layers.dcn.Cross(\n",
    "          projection_dim=projection_dim,\n",
    "          kernel_initializer=\"glorot_uniform\")\n",
    "    else:\n",
    "      self._cross_layer = None\n",
    "\n",
    "    self._deep_layers = [tf.keras.layers.Dense(layer_size, activation=\"relu\")\n",
    "      for layer_size in deep_layer_sizes]\n",
    "\n",
    "    self._logit_layer = tf.keras.layers.Dense(1)\n",
    "\n",
    "    self.task = tfrs.tasks.Ranking(\n",
    "      loss=tf.keras.losses.MeanSquaredError(),\n",
    "      metrics=[tf.keras.metrics.RootMeanSquaredError(\"RMSE\")]\n",
    "    )\n",
    "\n",
    "  def call(self, features):\n",
    "    # Concatenate embeddings\n",
    "    embeddings = []\n",
    "    for feature_name in self._all_features:\n",
    "      embedding_fn = self._embeddings[feature_name]\n",
    "      embeddings.append(embedding_fn(features[feature_name]))\n",
    "\n",
    "    x = tf.concat(embeddings, axis=1)\n",
    "\n",
    "    # Build Cross Network\n",
    "    if self._cross_layer is not None:\n",
    "      x = self._cross_layer(x)\n",
    "    \n",
    "    # Build Deep Network\n",
    "    for deep_layer in self._deep_layers:\n",
    "      x = deep_layer(x)\n",
    "\n",
    "    return self._logit_layer(x)\n",
    "\n",
    "  def compute_loss(self, features, training=False):\n",
    "    labels = features.pop(\"user_rating\")\n",
    "    scores = self(features)\n",
    "    return self.task(\n",
    "        labels=labels,\n",
    "        predictions=scores,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jDiRfzwVW9LH"
   },
   "source": [
    "### Model training\n",
    "We shuffle, batch and cache the training and test data. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:11:56.729400Z",
     "iopub.status.busy": "2020-11-30T12:11:56.728692Z",
     "iopub.status.idle": "2020-11-30T12:11:56.732337Z",
     "shell.execute_reply": "2020-11-30T12:11:56.732750Z"
    },
    "id": "qeFjmfUbgzcS"
   },
   "outputs": [],
   "source": [
    "cached_train = train.shuffle(100_000).batch(8192).cache()\n",
    "cached_test = test.batch(4096).cache()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5adSI3yOt2VQ"
   },
   "source": [
    "Let's define a function that runs a model multiple times and returns the model's RMSE mean and standard deviation out of multiple runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:11:56.738890Z",
     "iopub.status.busy": "2020-11-30T12:11:56.738296Z",
     "iopub.status.idle": "2020-11-30T12:11:56.739998Z",
     "shell.execute_reply": "2020-11-30T12:11:56.740409Z"
    },
    "id": "gTDk3GloquHO"
   },
   "outputs": [],
   "source": [
    "def run_models(use_cross_layer, deep_layer_sizes, projection_dim=None, num_runs=5):\n",
    "  models = []\n",
    "  rmses = []\n",
    "\n",
    "  for i in range(num_runs):\n",
    "    model = DCN(use_cross_layer=use_cross_layer,\n",
    "                deep_layer_sizes=deep_layer_sizes,\n",
    "                projection_dim=projection_dim)\n",
    "    model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate))\n",
    "    models.append(model)\n",
    "\n",
    "    model.fit(cached_train, epochs=epochs, verbose=False)\n",
    "    metrics = model.evaluate(cached_test, return_dict=True)\n",
    "    rmses.append(metrics[\"RMSE\"])\n",
    "\n",
    "  mean, stdv = np.average(rmses), np.std(rmses)\n",
    "\n",
    "  return {\"model\": models, \"mean\": mean, \"stdv\": stdv}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZRHjQ8g2h2-k"
   },
   "source": [
    "We set some hyper-parameters for the models. Note that these hyper-parameters are set globally for all the models for demonstration purpose. If you want to obtain the best performance for each model, or conduct a fair comparison among models, then we'd suggest you to fine-tune the hyper-parameters. Remember that the model architecture and optimization schemes are intertwined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:11:56.744202Z",
     "iopub.status.busy": "2020-11-30T12:11:56.743606Z",
     "iopub.status.idle": "2020-11-30T12:11:56.745420Z",
     "shell.execute_reply": "2020-11-30T12:11:56.745787Z"
    },
    "id": "Zy3kWb5Dh0E7"
   },
   "outputs": [],
   "source": [
    "epochs = 8\n",
    "learning_rate = 0.01"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Nz3ftiQLXdC0"
   },
   "source": [
    "**DCN (stacked).** We first train a DCN model with a stacked structure, that is, the inputs are fed to a cross network followed by a deep network.\n",
    "\n",
    "<img src='./assets/dcn-stacked-simple.png' width='50%'>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:11:56.751773Z",
     "iopub.status.busy": "2020-11-30T12:11:56.751084Z",
     "iopub.status.idle": "2020-11-30T12:12:16.110740Z",
     "shell.execute_reply": "2020-11-30T12:12:16.110227Z"
    },
    "id": "hiuYPJWhgw3J"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/5 [=====>........................] - ETA: 0s - RMSE: 0.9363 - loss: 0.8767 - regularization_loss: 0.0000e+00 - total_loss: 0.8767"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/5 [=======================>......] - ETA: 0s - RMSE: 0.9305 - loss: 0.8659 - regularization_loss: 0.0000e+00 - total_loss: 0.8659"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "5/5 [==============================] - 0s 15ms/step - RMSE: 0.9306 - loss: 0.8662 - regularization_loss: 0.0000e+00 - total_loss: 0.8662\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/5 [=====>........................] - ETA: 0s - RMSE: 0.9385 - loss: 0.8807 - regularization_loss: 0.0000e+00 - total_loss: 0.8807"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "5/5 [==============================] - 0s 3ms/step - RMSE: 0.9339 - loss: 0.8725 - regularization_loss: 0.0000e+00 - total_loss: 0.8725\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/5 [=====>........................] - ETA: 0s - RMSE: 0.9377 - loss: 0.8792 - regularization_loss: 0.0000e+00 - total_loss: 0.8792"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "5/5 [==============================] - 0s 3ms/step - RMSE: 0.9326 - loss: 0.8702 - regularization_loss: 0.0000e+00 - total_loss: 0.8702\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/5 [=====>........................] - ETA: 0s - RMSE: 0.9400 - loss: 0.8837 - regularization_loss: 0.0000e+00 - total_loss: 0.8837"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "5/5 [==============================] - 0s 3ms/step - RMSE: 0.9350 - loss: 0.8750 - regularization_loss: 0.0000e+00 - total_loss: 0.8750\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/5 [=====>........................] - ETA: 0s - RMSE: 0.9390 - loss: 0.8817 - regularization_loss: 0.0000e+00 - total_loss: 0.8817"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "5/5 [==============================] - 0s 3ms/step - RMSE: 0.9339 - loss: 0.8729 - regularization_loss: 0.0000e+00 - total_loss: 0.8729\n"
     ]
    }
   ],
   "source": [
    "# TODO 3: Here is your code.\n",
    "dcn_result = run_models(use_cross_layer=True,\n",
    "                        deep_layer_sizes=[192, 192])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZwTn_UpDX_iO"
   },
   "source": [
    "**Low-rank DCN.** To reduce the training and serving cost, we leverage low-rank techniques to approximate the DCN weight matrices. The rank is passed in through argument `projection_dim`; a smaller `projection_dim` results in a lower cost. Note that `projection_dim` needs to be smaller than (input size)/2 to reduce the cost. In practice, we've observed using low-rank DCN with rank (input size)/4 consistently preserved the accuracy of a full-rank DCN.\n",
    "\n",
    "<img src='./assets/dcn_lowrank_simple.png' width='50%'>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:12:16.118040Z",
     "iopub.status.busy": "2020-11-30T12:12:16.117229Z",
     "iopub.status.idle": "2020-11-30T12:12:27.987171Z",
     "shell.execute_reply": "2020-11-30T12:12:27.986651Z"
    },
    "id": "NYxbHI7ZNJX7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/5 [=====>........................] - ETA: 0s - RMSE: 0.9361 - loss: 0.8763 - regularization_loss: 0.0000e+00 - total_loss: 0.8763"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "5/5 [==============================] - 0s 3ms/step - RMSE: 0.9307 - loss: 0.8669 - regularization_loss: 0.0000e+00 - total_loss: 0.8669\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/5 [=====>........................] - ETA: 0s - RMSE: 0.9391 - loss: 0.8819 - regularization_loss: 0.0000e+00 - total_loss: 0.8819"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "5/5 [==============================] - 0s 4ms/step - RMSE: 0.9312 - loss: 0.8668 - regularization_loss: 0.0000e+00 - total_loss: 0.8668\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/5 [=====>........................] - ETA: 0s - RMSE: 0.9353 - loss: 0.8748 - regularization_loss: 0.0000e+00 - total_loss: 0.8748"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "5/5 [==============================] - 0s 4ms/step - RMSE: 0.9304 - loss: 0.8667 - regularization_loss: 0.0000e+00 - total_loss: 0.8667\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/5 [=====>........................] - ETA: 0s - RMSE: 0.9376 - loss: 0.8791 - regularization_loss: 0.0000e+00 - total_loss: 0.8791"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "5/5 [==============================] - 0s 4ms/step - RMSE: 0.9334 - loss: 0.8718 - regularization_loss: 0.0000e+00 - total_loss: 0.8718\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/5 [=====>........................] - ETA: 0s - RMSE: 0.9364 - loss: 0.8769 - regularization_loss: 0.0000e+00 - total_loss: 0.8769"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "5/5 [==============================] - 0s 4ms/step - RMSE: 0.9300 - loss: 0.8658 - regularization_loss: 0.0000e+00 - total_loss: 0.8658\n"
     ]
    }
   ],
   "source": [
    "dcn_lr_result = run_models(use_cross_layer=True,\n",
    "                           projection_dim=20,\n",
    "                           deep_layer_sizes=[192, 192])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5O5AoNOdaQ80"
   },
   "source": [
    "**DNN.** We train a same-sized DNN model as a reference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:12:27.998337Z",
     "iopub.status.busy": "2020-11-30T12:12:27.997644Z",
     "iopub.status.idle": "2020-11-30T12:12:39.010253Z",
     "shell.execute_reply": "2020-11-30T12:12:39.009771Z"
    },
    "id": "iBPpwD4cGtXF"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/5 [=====>........................] - ETA: 0s - RMSE: 0.9552 - loss: 0.9124 - regularization_loss: 0.0000e+00 - total_loss: 0.9124"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "5/5 [==============================] - 0s 3ms/step - RMSE: 0.9460 - loss: 0.8985 - regularization_loss: 0.0000e+00 - total_loss: 0.8985\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/5 [=====>........................] - ETA: 0s - RMSE: 0.9417 - loss: 0.8868 - regularization_loss: 0.0000e+00 - total_loss: 0.8868"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "5/5 [==============================] - 0s 4ms/step - RMSE: 0.9358 - loss: 0.8773 - regularization_loss: 0.0000e+00 - total_loss: 0.8773\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/5 [=====>........................] - ETA: 0s - RMSE: 0.9402 - loss: 0.8839 - regularization_loss: 0.0000e+00 - total_loss: 0.8839"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "5/5 [==============================] - 0s 3ms/step - RMSE: 0.9392 - loss: 0.8842 - regularization_loss: 0.0000e+00 - total_loss: 0.8842\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/5 [=====>........................] - ETA: 0s - RMSE: 0.9401 - loss: 0.8838 - regularization_loss: 0.0000e+00 - total_loss: 0.8838"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "5/5 [==============================] - 0s 3ms/step - RMSE: 0.9362 - loss: 0.8772 - regularization_loss: 0.0000e+00 - total_loss: 0.8772\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/5 [=====>........................] - ETA: 0s - RMSE: 0.9408 - loss: 0.8852 - regularization_loss: 0.0000e+00 - total_loss: 0.8852"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "5/5 [==============================] - 0s 3ms/step - RMSE: 0.9377 - loss: 0.8798 - regularization_loss: 0.0000e+00 - total_loss: 0.8798\n"
     ]
    }
   ],
   "source": [
    "dnn_result = run_models(use_cross_layer=False,\n",
    "                        deep_layer_sizes=[192, 192, 192])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cBY0ljpl3_k5"
   },
   "source": [
    "We evaluate the model on test data and report the mean and standard deviation out of 5 runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:12:39.015313Z",
     "iopub.status.busy": "2020-11-30T12:12:39.014644Z",
     "iopub.status.idle": "2020-11-30T12:12:39.016997Z",
     "shell.execute_reply": "2020-11-30T12:12:39.017413Z"
    },
    "id": "a1yj3pp0glEL"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DCN            RMSE mean: 0.9332, stdv: 0.0015\n",
      "DCN (low-rank) RMSE mean: 0.9312, stdv: 0.0012\n",
      "DNN            RMSE mean: 0.9390, stdv: 0.0037\n"
     ]
    }
   ],
   "source": [
    "print(\"DCN            RMSE mean: {:.4f}, stdv: {:.4f}\".format(\n",
    "    dcn_result[\"mean\"], dcn_result[\"stdv\"]))\n",
    "print(\"DCN (low-rank) RMSE mean: {:.4f}, stdv: {:.4f}\".format(\n",
    "    dcn_lr_result[\"mean\"], dcn_lr_result[\"stdv\"]))\n",
    "print(\"DNN            RMSE mean: {:.4f}, stdv: {:.4f}\".format(\n",
    "    dnn_result[\"mean\"], dnn_result[\"stdv\"]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K076UbT1nnq3"
   },
   "source": [
    "We see that DCN achieved better performance than a same-sized DNN with ReLU layers. Moreover, the low-rank DCN was able to reduce parameters while maintaining the accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eSF0gNLGX1Za"
   },
   "source": [
    "**More on DCN.** Besides what've been demonstrated above, there are more creative yet practically useful ways to utilize DCN [[1](https://arxiv.org/pdf/2008.13535.pdf)]. \n",
    "\n",
    "*   *DCN with a parallel structure*.  The inputs are fed in parallel to a cross network and a deep network.\n",
    "\n",
    "*   *Concatenating cross layers.* The inputs are fed in parallel to multiple cross layers to capture complementary feature crosses.\n",
    "\n",
    "<div class=\"fig figcenter fighighlight\">\n",
    "<img src='./assets/dcn-more-simple.png' width='50%'>\n",
    "  <div class=\"figcaption\">\n",
    "  <b>Left</b>: DCN with a parallel structure; <b>Right</b>: Concatenating cross layers. \n",
    "  </div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GEi9PtCEdyma"
   },
   "source": [
    "### Model understanding\n",
    "\n",
    "The weight matrix $W$ in DCN reveals what feature crosses the model has learned to be important. Recall that in the previous toy example, the importance of interactions between the $i$-th and $j$-th features is captured by the ($i, j$)-th element of $W$.\n",
    "\n",
    "What's a bit different here is that the feature embeddings are of size 32 instead of size 1. Hence, the importance will be characterized by the $(i, j)$-th block\n",
    "$W_{i,j}$ which is of dimension 32 by 32.\n",
    "In the following, we visualize the Frobenius norm [[4](https://en.wikipedia.org/wiki/Matrix_norm)] $||W_{i,j}||_F$ of each block, and a larger norm would suggest higher importance (assuming the features' embeddings are of similar scales).\n",
    "\n",
    "Besides block norm, we could also visualize the entire matrix, or the mean/median/max value of each block."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-30T12:12:39.025241Z",
     "iopub.status.busy": "2020-11-30T12:12:39.024485Z",
     "iopub.status.idle": "2020-11-30T12:12:39.238442Z",
     "shell.execute_reply": "2020-11-30T12:12:39.237895Z"
    },
    "id": "47ibaEBJxOoe"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/ipykernel_launcher.py:23: UserWarning: FixedFormatter should only be used together with FixedLocator\n",
      "/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/ipykernel_launcher.py:24: UserWarning: FixedFormatter should only be used together with FixedLocator\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 648x648 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAE8CAYAAAABo4xnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAA+E0lEQVR4nO3dd7wcdbnH8c83BQIEEkgA6eECSidAIIh0lE5ClSIIWAKKcBURBFSaevXqVYoFEekICFICUgQCUgSBUEJXkBqlhNBryvf+8fttznI8Zeec2bObs8+b176yOzs78+wmzDO/LtuEEEIIPTGg0QGEEEKYe0USCSGE0GORREIIIfRYJJEQQgg9FkkkhBBCj0USCSGE0GORREIIIfRYJJEQQgg9FkkkhNAyJMU1r2Txg4YQWoKk+YC1JQ2WtJGk9RsdU38QSSSE0CoWBHYAfgdcAMxqbDjNSZKK7B9JJITQ70mS7ZeBO4BxwDXA442NqvEqCUPSGpJWkrSCbRdJJJFEQgj9Xr4wbgYsCYwHDBws6eMAkoY2LrrGyb/LtsDvScn1dkmruMDMvIPqFl0IITSXNYD3bf9F0nvAwcCHkjYHxkva1/arjQ2xb0laFvgOKYGsB7wITCtyjCiJhBD6NUmLSxoGvAeMlTTA9t3AL4ARpJLJ71otgWTTgWuBDYDDgM/afkXSOEmL1XIAxXoiIYT+StIKwInAKsBfgc2AY0ltI+/Zfl3SUNtv53aTlrggSloRGAXcDNwCrAMMtz1D0ljgp8CXbD/R7bFa5DcLIbSI9skgNxKvBQwH/ghcRbqAjiT11nq2FZJH5XeR9CngUGA+4EhSaeTPwK3AU8B+wHG2r6zpuC3w24UQWkTVhXJH4DPAEODHtp/KjedXAHvlKpuFbb/WyHj7mqTPAD8DTicl0MdI3Z2fJCWWN4EHbU+qtWQWSSSE0K/kXlg/BXYnlTweA/bPVTW/Bi63/edWqr4CkDQY+Alwn+1zJY0C9idV9f3A9pSeHDca1kMIc7UOxjSsDXwX+DjwAXBUTiAi9T56A1L31j4NtMFszwBeBraRtIjtZ4DzgZWBPSQt3ZPjRhIJIcy1JA0BxubnK0taHXgd+BJwDLCP7eck7Qd81/bxtv/WsID7UNVAwpUkrZk3Xwj8G9gz/3YGXgE2Bj7Zk/NEEgkhzM0WA8ZIuhi4DHgJuAdYAbgYeDHPkXU4cG/DomyA3Da0PXAl8HVJt5C6Of8FWI3UK+tK4CBStV+PSiIx2DCEMNfKpYwBwC7AebZfAV6RdATwVdId9hLA0bavaaV2EEnrAT8EtiYNJPw9cDap6+7EXDp5BViVNPByxx6dp0V+zxBCPyVpIWAP0p20gJNsT5e0EvAvYCHb/26lBAIgaQSwFKm09gNgS1IbyMeBrW0/L2kZ4FTgGNuP9OQ8UZ0VQphrSRpo+03bvyUNIFwQ+EqeD+pAYIDtf0NrNaTnhPlq7nG1MXCl7bdJbSIzgUUAbD9PajfqUQKBSCIhhLmY7erp3G8EriYNKvwV8BfbbzUirkZrlzCfAFaTdDRwCPAF2w9WFujKyaXHojorhDBXynNgzc7P249SX9L2v1qhCqu77yjpv4BNSdVZF9u+qtTz9/PfN4TQD1SNRF8CmGF7Wt4+sF1p5CMX1Y7e748k7QCsbPunNexbamKN6qwQQtPLCWQHUvfUX0o6JW+fJWlgZb+cNCxpqKSlWySBrAN8nTT3VWf7DK48L7roVHciiYQQml7uabU78GXgW8Dqks6AtkRSKXVIGk6aZHHRhgXcR/J07QeReqDdnbcNaLfPwDxif2FJR0G5nQwiiYQQmpakAZKWAs4BFiDN+/QcsCswStIFkBJJTiDDSAPnvmf7/oYFXkftShHTSQMG35T0LQDbsyuJpF1ivQQofbR+JJEQQtOpXChtz7Y9lTSh4hLAxpLmybPvfhZYrjKlRx4vcj1wrO3bGhR6XVW1DW0t6XDgK6SR56eSfotDoS2RVCWQPwDH255UdkwxYj2E0FSqLpSbkhaRmgpMBN4lrX9hSZPygMLNbM/MSWcz4Ou272pU7PWWf5ctSbPxfpnUrXkoKYkY2FnSYbZ/lhPJEOBPpBH7dUms0TsrhNB0JG1NulBeAgwDNidVYa1KWpnweODadt16B9me2YBw+0TuQCDgJNIaIIOA/wN2zaPPBwDbA0/bfjh/ZkFgpO2n6xVXlERCaHHNMJYiNxCPqjQOAxsBP7L9+/z+BNJ0JjtJWhJ4tX3M/TWBVP39zGP7PUl/B75AWgdkr5xAJgBv2L64+rN5sGVdB1xGm0gILaqqgXZkfj24i93rGccgYBwwPd85A8xPGiBX8SfgLaX10M9openccxXWVsBJubTxKmnVxuOcVmxci7QqYUNWaYzqrBBaUNXFaXvga8D9pIvTb22/2YA4BpHmczqC1Lvq76RG8j/ZPlbSWOAXwGfrWTXTjCRtR6q2OsT2jXnbBGAnUjvRMqSVCSc2JL5IIiG0JknrkqYGHwd8n3T3v4/td/ro/EOAFW0/nKfmWArYBpiXvBYIqfvq48BawJG2r+6L2JpFLnmcQuri/BCwLbA3cBwp6S9ImmTyiUZVS0YSCaFF5Av1Krb/lF9vT5o+/WHgZ8Cetp+W9AngH5V5qeoYz6qkapmVSGtejCGNBfkCsDBwLvAIadDgfLb/2QztN31F0rKkRbYOAyYAU4AHSR0NNgC2yV2dGyoa1kNoHR8HLpT0uTwJ33PAj0jXgU1tv5wTyzjgm0CvZnftju1HJe1F6qp6ku03gDcknUe62z4QuNz2DVWfqVsCaabeXZIWJ43Mf9L2/0h6EHgqlziWJCXYhrRhtRcN6yG0gDxy+Trg28DJknax/RBpepAbgdGSNiItXvSn3k4P3k0s1SOuzwKOJo39mCBpYdvPApeT7sKfq1cc7WLaAhif22Yaot3v8jJwF/Bfkg4Gbs0J5HPAtcCvbL/ciDjbi+qsEFpEnsBwT2AIaSnUXUkN6lsD+5JWAbzYaenUulQbtRtIuDipq+5Nkj5LGix4B/A6qYrrPNuvlh1DBzF9nNRwfYzTIk4NI2lDYCXb5+SkshvwKdKaIGcB+wAv2r66War2ojorhH4uX4wWBU4EjrB9g6RxwJnABNtn5jmoZPv9el6ccgLZETihcn5J69n+kaRZpG69O5F6ItU1geRG61HATcA1tqeo8VPHLwh8T9Is2+dLuow0wPKLpAb0X0JzjO2piOqsEPo5Jy+TSh2v5AvlRODHwGWSdrT9ge33K/vXKxZJS5MaiceRehcNBzaQdILtP9o+FNjY9pXtqnfKjKF6Xq5/kjoVfFrSanmuqbqctxa2ryfNynu4pM/nhHYz8GT+s7JfUyQQiJJICP1SVbXRSGCm7deBN0l3tIcBs4A7SWMx3ujD0N7K5/8YaUzIeGAd4OeShtn+79wmUpcLZdXvshmwHvAU8EvSvFMXStq9kd1lAXJJ8XDggly9tRWpxPhoI+LpTiSREPqhfKHcjlRtNFXSdOAbpF49Z0l6GdgOOMD2nX3QBjIamElalfAJSdsAV9l+VtJqwKXARWWfv72q6rRjgdNII71Xt32CpPmB6yRta/vxesUgaXlghO17u4jzRkmbkKZ/Oc/2HfWKp7ciiYTQD0lahTSA8GvAA8CFwM+BXUgN6UsAV9u+E+p+178FcD5wDWkq9+OAvwK/kWRS9dZeXV1UexnHMFLyejf3vvo0KYGuRxpgeRaA7R/mqqzFSAMc62Vz4CeStrF9T0cJXGka9ydIDeqVbU3TDlItemeF0A9JWoE0qvlLtj/I224F/mD7F30YxwakhvJrbN+auxFfSUpks0mNxlNt39z5UXp1/vlICfRW4HTgHdL0KR8jdTbYLw+w3AF4pxJHPS7YVW0xzol0F1I11V2dJJKmGbfSlWhYD6EfqFygJG2aq44+JF0ox1TtdiHwQR+H9k3SCPRZ+UJ5O3A46eJ9n+3z65hAZPs9UtXV1sAe+UJ9KW1diJ+WtDGplDaj8tl63PHnDg6WtC2wHKljwfWSNszb5zTo584PM5WWtD247FjKFEkkhH4gX4S2JzUSL2b7eVLSOEXSVyV9iVS11SeTF0r6hKS1bO9O6kL7ddKcWJAS3OJqtxZ4PcJo9+dJko4g9XT6Gal78bmktpGv5wRX34BS77T/A86yvTlpoOWl1YlEbUvaDiMNunys3nH1RrSJhNAPSFqYtOrfl6vaOc7MDeqrk+58v+E8C2wd4xgADCSVPhaS9Evbe0maCPxF0uWkSRZPcZ3n5nJa2W8safLCPYCVScnstbztJtI0+O/bfqyP2hxeASYDL+Rk8UtJKwKTJG2U24Wq10T/Tl8kt96IJBKantIaEnWdx6kfGESaS+lJSDPk5nEf19q+og8bZeez/Y6knwOHAF+QdJbtcZJ+Txoxv7fTvFkD6p1ISFV6D9h+EHhQ0lRSddbiwM9yiQ2oe+eCEfkcr+Zqqz1s/0/e7SpSI/+8+TPzkRLcN5o9gUBUZ4UmJ2kB4BpJ+zQ6lmZmu3KHe7ikhZxGnm8OTMyllLr/v567rl6Yq7FeJE1hPgM4StKqtvcGngF+mO/CS08g1e0K2T9J83J9XNK8+aJ8CbAzaaBjXeUEsgOpM8GvJf0YOArYSdIpkr4PnAwcVtWN931Skrm13vGVIZJIaGpOa1v8L/DN3L8/tFN14Tw3/3mtpAOBXwGn2n7NdZrKo6pBf0HgPeAe4LuSVrf9Eqn+fz3gy7nksRNpvMjH6hFPvmhvI+loSQc6TTL5PKkaa2el8SkrAl+z/a96xFBN0nrAMaQS2G3A7rn0szNwL6m32DedlwXOv5FtP1nv2MoSXXxD06qqClgD+G9gY+Bwp2nMQwdyY+y+pLvZf9qeVK+qrKq/nx2Bz5Om65g3n399UhvNB8BvgKNtP1B2DB3EsjZp0OLvSOMxXrD9ZUlfAdYgtYv8vJ7/hqp/b0ljSO1Rg0iDPffKPcJWdtWAxj6sbixdJJHQ1CRtTepxdCKwJmkRoxNsX9rQwJpELRefel6glEZVn0yqv78lb1sc2A84gDS9ylF9kfglrZ/Pe4ft3+e2hT8C/7b9xbzPwrZf64PEulXeNI000PJ1YCvbbysNvvwGaZDli3Nr8qiIhvXQVPIFaDXbk/KmtYCfOk2NPZA0cd/xkmbYvrJhgTZIvlCuQFoq9Tnbb7a/IKrdTLR1vkitBFxg+xZJ89l+L1dj/a+kK4DZfVg18zHSLMBvS7rCaYT6LqTqvSttjyfNH1avRvRK19zNSMnrCdtjJF1KWmRrFUkrk9Z0OdL2v8uOoRGiTSQ0jVy/viXwfK6WgdTHf0eAfGG8Hfg3adqIxTpoSC0jjqa6ucrdZskN5eeQ1gE5GviOpMU7SiCShknaXiWPxahqAxmVNy0IbAHgNLAPSZ9UmhH37/VMIFWxrKY0/uImYH/S0rHb5qT2Pmld8hNzjKW3DeXfekj+3ceRBi4eDNyXz/k90tQq+5JK0oc5rwdSdiyNEEkkNI18MbwYmA6cmOvaf0q6XpyTd/svUkPpTrZfLvuOUtLHSHfR/1XmcXsYy/wwZ7zDmqSR13vb3g04ldTusZPaVA9Sux54o+weULmqZjvgVEnLkaqy5pN0tqQF8134+cAiZZ63m1jOIyWPSaTeXz8ntc/sXEkkru+8XF8GFpY0mDQm5zDb5wKfkLRE3vV/naa5/6LTdO9NNZ17b0QSCU0l3ynOAKaSSiWfIfVsGSnpStIF42rXb1rsV4ElgUOVuqw2RO7tdJ1S91xIc0ytR77rJy2d+hywodtUEshlpA4IpY8xUJpS5f+AH9p+Nl8ItyeNu/gd8EPgUNu3lX3uDmIZRUqs40gDCE26Nl9Bmh/rq8Cwzj5fkjeBC/LzfYFf2L5Z0kLAwsD7ud3oRqVp+Ru54FVdNFWxPYRcv/+mpF+R1r7YjjQD6/b5f8yFbL9Qj4ZRpQnvZkg6lTTGYQVJX7H9QpnnqYXtt5SWjF1S0jq2L5I0BNhX0tT8+jFgb0mL2J4uaSh5ipF6JJBsJeB25+njgUG23yVVHw0GFnbfrf39HnADaX6wfYB9nQbzfcZpUas7bE+rx4klzQssaHuapA+B3UnrosyWdGn+N/xn0g3QPqR2vbrE0miRREJTyVUUyhfRM0nVFHtKGm77j9SpYVSpf/5MpXEEx5KSyH8D35d0rPNCSX2hqmH8LWBd4CpJW9g+W9IM4Ke57n0kqbvq9KqPT7B9X4mxtE/WTwOL5DaPR4AZua1madvnSXqlrHN3EdMoYCFS54LNSO0Py+UL98aktqInbddlnjClDh7rA8vmG5t1Sd2Z3yMlkoGkUtn8pJ6F29q+vh43Ps0guviGplH9P1nlef6f9Euk6TtKn4guV1kNsP1UboT+LTDF9sn59fmkC9ZXbT9X9vk7iKfyvTclVQvtKmlP0jTm43IvqANIc0HdaPun+XOlTxteFcvGwCeAd0mDCb8GPEuqcnyGdMH8qus4wroqlg2B40ntZt8krf1xAmlk+j1527Guc889SauSqvVGk7own52Ty36kHoVT8mO27cn1jKXRok0kNERVz5oFcjVIpRQyoOq5bL9JGnX9WPXnSvQpYETuXTMbeAQYrjR1yGxSA+0ngW8pjTuoq6rG4t8BG0la2vZFwIHAH3NVzVnAGcB4SePz50pfdyLHsgWpHWoh0hTu25NGWs8idVs9EjimngmkKpatgJOAP5Gq1b6a45iQd1sSOMJ9sz77o8DdpCWGF1IaoT/L9pmkBa1WI40BmVz9uf4oSiKhz1XdVW5PmqTvaeAl28d1sO+gXM00LzAw17+XHc9IUlvCnsAQ0p3tqaSL5aLA/wA/cZ4dt57ynfavciz7AZe7bUqMvUnJpdLjZ3PgXldNIlhiHCJVy5wC3JmrqpYFvkUaBf7jnPAXtP1GPatqciyDSYnzRtvn5iqt75FuhI9yH4y5qPp3O5rU9bzS9vMt0qDC03Kcy5N+o7qXXJtBlERCn6m6i6tUkRwPHEaaHnt89Z2+ksrCPMNJ/exH1iOu3OD5B+BM4CXg16S2mLOAK4Df9kUCyRYntWs8TrrbXg3mVJ9cBixr+3XbrwNX1COBwJwFlGaSeoCtndukniN16f2spCVsz7b9RmX/esRRFcuHwKPAakqjzp8BfkxKpPtXSrP1lP/dbkNap2Ur4B/APKRZgYeTqrceybu2RAKBSCKhjyiNRB+vtsFvQ0l3cMuT/ofcyfZ7SmuDQyolV3dZ/U09/8e0/YN8nitIDbb7AkcAu9m+tl7n7SCOOSUPUknoRUkrkRLaak6z9fblXEv3k+6uN1HqHSZSg39fr5AI8CCwAGmd9nlIEzn+nfR3Na7eJ5e0JGm0+Q6k3+U54C2nnnCnkObsGt+HNxxNIXpnhb4yhnQnOTx3iRxEqmt/DviMU2+sLYHdJR3lNL/RMNJaC0e7xC6rnV2Abf9E0ixSt9HP2p5S1jmLxpI9S5rBeCapsXhOA209EkhHseReRf9Fags5hDTu4sf+aI+w0nUSy7W5I8T2wKGkar1xpIv6AnWKozKAc2FSz8CJpLFLBwA7OnXx3YXU7fnaqs/1y55YHbIdj3j0yYPUOPsr4MD8+n9Jd5cjSBeDR4Dt83sDSQPXNirx/JU2wOHd7PetMs/b01hIF6vngC2qP9NXsZB6rVWeL0aaBfcTTRDLx0jdapfNv9HjlbhKjGN41fN1SFVWSwHXkao8B+f3xpB6ha1Zz38vzfxoeADxaI1H/p9tUVLX1F+SGo3nyYnkj8DVpP701Z8ZVoc4tiZNj7FQdxdCUtVNXS6WtcYCrF4Vy4C+jqWLuBoeC2mK9TvKvoCTqu/+TioBV5LWGfn5UqQS4snA94EHSFVYdfkt5oZHVGeFvvIVYH6n9bYHkLrWzrJ9BKR5opx7Xikvm+rcaFsWSauRxhF8y3n22w72qVRflD7uokgsVb/Bw5IG255BmtajT2Nx5erd1ktuoFNX1rosa1skFqcBoJ8qOwanWQt2J83+O4M0dmh2/u5TJX2CVJ31Dmksz60tVX3VTjSsh7ro4AL9TeAtpZHOF5L62G8maYLSrLnvV3asxwUq9/zajrQmyTL5PG63TyWBDCfNW1WXSQRriYVU8iDHcm0jY2nXS+76JollEUk3Kq9dXmIMlR6ED5J6fn2b1K16BGk+tUOAzwLTbJ/rPD6mVRMIRMN6qBPbVpoOYzBwv+1XJL0F7AY84tTXfwBwd73u+Kv69c9HSlK/Jo0D2VHSK25b07p9AvkD8H2X2HgcsZQey0U5llfrEMvWwGinsTAbkjpazENafngj0vT355d13rldDDYMdSNpD9Ikiv8gra1wDWmK8oNs/7WPYhhPmgBvCGmZ1n+QevOsAFzkqpHWuQfOH0grJ5Y+C23EMlfEsjlpipkvVs4raUXgFuB7TiPSQ7VGN8rEo/88aLspWZc0VciKpCrTDUg9WP4PeIxURVC3htmqeDYE/kpq0D8fuDlvHwUcQxoBPTxvG0C6eGwRsbReLKTegANJo86/WLVtUH6+Omm6+ZUq2+KRHlESCaXKVQGnknpb7Uka3/DbPKJ4B1J98gW2r+6DWPYEZpPmV/om8DnbTytNmT4vqffXP6v2X9D2WxFL68WivLSvpBNIa7L/utKhQdI6pMGF87kO0+7M9RqdxeLRPx6ku8RhwI3AdnnbOsBT5Du7dvuX3nWWtpLQ8vnPcaR+/bdWbdudVB0ypM6/R8TS5LFUxbQCqZp1KdJSzLeTSxykGXnvAVbui1jmxkf0zgq9UtULazBpOoy/Ae/lRtD7SGtybK92cxs5/99bJtuWtC1wTh7ZfAupsfYxYIikz5DWCjnHae3tuolYmj+WKm+QShqnAjcDvyetjHg+qTrth05zmYUORHVW6DVJO5HWmX6K1G//BuAHbpvK5CBgL9dx3EWOY13gEmAP2/fkbfMDPyL1qBkBnGb7mnr3649Y5opYVrT9ZH4+gjSVymhSI//8pFH6s20/0srjQLoTSST0SFV3yOHA2aTqB9M2qeK5pOqAjYDv2p5Y53iWI82ptFA+9+6kifkeAw5wHufgNPttXUUszRtLVTfhAaRFoy6xfXx+b1FSCWQYaWqePlvNcm4W1VmhR3ICGQvsBUy2/XunQYTHkVaZG0BKLl+1PbGq2qt0kj5GWqBoFVLj7K9ICW1v0tTqm+Zd36xXDBFLc8ciaUGAnEA2Ic25dTJpGvmD83uvkNo/3ib1Fgs1iMGGoZCqEsiGpOnJnwQWk3Q7aSbTiUpL2h5DWizobajLmujV1QsvkybhG0G6OP3W9gtK01N8DHgxx1CvqToiliaOJVeX/UnSyaRJPn9DGrf0PPBv0prsQ3MMe5BKRQ+XHUd/FdVZobBcAvk+cJjthySdSFqU51Lgr07dIpeyPbXOcWxI6tFzgdL61nsA65GmnH+HVLV2vO0r6hlHxNL8sUjamTQ+6R3gO7b/mgcRbgOMJZWAFiStJHl5PWPpb6I6K/TEMNK8Qp/Jr08AppNm5t0IoN4JJFsYOFHSnrZnkdpl3gQ+T+pe/A3bV9SzKi1imTtiyYnhGNJA2C3y5mdJU+2/DHwN2M/25X30u/QfZfQTjkfrPYDxpKqBvfPrQaT1P1br4zi2Ja1JUoljM1LvnxUa8JtELM0fy86kKti98utNSe0gi1HHaf/78yPaREKP2L5SaZrsEyXNY/ts4OgGxHGtJAPn5qqT7UhrlD8VsUQsHcRyuaSZpHEqe5LGqJxg++W+jqW/iDaR0CuSxpH6+H8aeMmpyqIRcaxOmofpUZe4lG7E0m9j2YVUDftl23fGOJCeiyQSek3Sok7dI0OYa0haxHVeK74VRBIJIYTQY9E7K4QQQo9FEgkhhNBjkURCCCH0WCSRUBNJExodQ0XE0rFmiaVZ4oDmiqW/iiQSatVM/zNGLB1rlliaJQ5orlj6pUgiIYQQeiy6+LaA4YuM8BJLLdurY7w2fRoLLzKy17E8//p7vT7GzLdfZ9DQ4b0+zmILztPrY7z12nQWXHiRXh9n6DyDu9+pG9NffYVFRvR+BvPezhz16rRpjBjZ+38rAL29PJX1m7zw/LNMf3WaAAYutJw9s7Z/x37vlettb9PrAJpYTHvSApZYalnOvPzmRocBwLeueKjRIczx1S2Wb3QIc2y0XDkX3TIMHNg8FRSzZtVllvrCdtjyU3Oee+b7zLvynjV97v37T22ev9g6iSQSQghFiN4X1/qRSCIhhFCUmqe01miRREIIoagoicwRSSSEEApRlESqRBIJIYQiBAwY2Ogomkak0xBCKESpOquWRy1Hk4ZLulTS45Iek/TJdu9L0imSnpQ0RdI6dflaPRQlkRBCKKrc6qyTgets7yZpHmD+du9vC6yUH2OBX+c/m0IkkRBCKKqkhnVJw4BNgP0BbH8IfNhut/HAuXnlxbtyyWUJ2/8uJYheiuqsEEIoQkptIrU8YKSke6se7efyWh54BThL0v2SzpC0QLt9lgKer3r9Qt7WFKIkEkIIRdVenTXN9pgu3h8ErAMcYvtvkk4Gvg18t5cR9pkoiYQQQiG5i28tj+69ALxg+2/59aWkpFJtKrBM1eul87amEEkkhBCKGqDaHt2w/SLwvKRP5E1bAo+2220i8PncS2sD4I1maQ+BqM4KIYRiRNm9sw4BLsg9s/4JHCDpIADbpwHXANsBTwLvAgeUefLeiiTSh/I/jHdtn1vG5ySNAq62vXp5UYYQuqZSBxvafgBo325yWtX7Bg4u7YQliyTSh/JdRZ99LoRQJzF31hzRJtIJSaPyCNKzJf1d0gWSPi3pDkn/kLS+pEUkXZFHkd4laU1JAyQ9I2l41bH+IWlxScdJOjxvW0HSdZImS7pN0spdxFL9uXUlPSjpQZr47iSEfq28hvW5Xmt8y55bEfg/YOX82BvYCDgcOBo4Hrjf9pr59bm2ZwNXAjsDSBoLPGv7pXbHPp3UrW/dfLxf1RjTWflza3W1k6QJlb7pr02fVuOhQwjdqnXKkxYprUQS6drTth/KieER4KZcP/kQMIqUUM4DsD0JGCFpIeBiYI98jD3z6zkkDQU2BC6R9ADwG2CJ7oLJpZvhtm/Nm87rbF/bp9seY3tMGcvahhCq1D7YsN+LNpGufVD1fHbV69mk325GJ5+7E1hR0qLATsD3270/AHjd9ujSIg0h9JGYCr5a/BK9cxvwOQBJm5FGp76ZSyuXAz8DHrP9avWHbL8JPC1p9/xZSeqyeip/7nXgdUkb5U2fK+l7hBCKiOqsOaIk0jvHAWdKmkLqv71f1XsXA/eQJ1brwOeAX0v6DjAYuAh4sIZzHpDPaeDPPQs7hNBj5Y8TmatFEumE7WeA1ate79/Jezt18vl7Sf/cqrcdV/X8aWCbGmOp/txkoLrUckQtxwghlCWqs6pFEgkhhKJapNG8FpFEmoikY4Dd222+xPYPGhFPCKETLdLeUYtIIk0kJ4tIGCE0M0V1VrVIIiGEUFSUROaIJBJCCAUIGDAgSiIVkURCCKEI0a7fZWuLJBJCCIUIRXXWHJFEQgihoEgibSKJhBBCQZFE2kQSCSGEIgSqYf30VhFJpAU88dRUttj9O40OA4Anb/5Zo0OYY1ATXQiaKBTmm6d5RmPPmOVGhwDAwKq/IEWbyEdEEgkhhIIiibSJJBJCCAWVmUQkPQO8BcwCZtoe0+79zUirpT6dN11m+4TSAuilSCIhhFBEfdpENrfd1TrWt9neoeyTliGSSAghFBTVWW1i7H4IIRRQaViv5QGMlHRv1WNCB4c08GdJkzt5H+CTkh6UdK2k1er25XogSiIhhFBQgZLItPZtHB3YyPZUSYsBN0h63PatVe/fByxn+21J2wFXACsVDrpOoiQSQghFqcZHDWxPzX++DFwOrN/u/Tdtv52fXwMMljSyjK9RhkgiIYRQhNIsvrU8uj2UtICkBSvPga2Ah9vt8zHloo+k9UnX7VdL/149FNVZIYRQUIkN64sDl+fjDQJ+b/s6SQcB2D4N2A34iqSZwHvAnrabYxQmkURCCKGQMkes2/4nsFYH20+rev4L4BelnLAOIomEEEJR0cN3jkgiIYRQhGJlw2qRREIIoaAYbNgmkkgIIRQVOWSOSCIhhFBQlETaRMVek5O0pKRLO3nvFkndjYYNIZRIUmnjRPqDKIk0CUmDbM9sv932v0j9xEMITSJKIm0iifSQpFHA1bZXz68PB4YC04GDgJnAo7b3zCNRTwVWBwYDx9m+UtL+wC75cwOBTbs6j6T5gLNI/cofB+brIr4JQJrMbfDQ3n/hEEKbyCFzRBIp37eB5W1/IGl43nYMMMn2F/K2uyXdmN9bB1jT9vQajv0V4F3bq0hakzQxW4dsnw6cDjBg/sWaZnRrCP1BlETatEalXd+aAlwgaR9SaQTSfDjflvQAcAswBFg2v3dDjQkEYBPgfADbU/K5Qgh9SRSZCr7fi5JIz83ko0l4SP5ze9LFfkfgGElrkAq/u9p+ovoAksYC7/RBrCGEkggxoPyVDedaURLpuZeAxSSNkDQvsAPp91zG9s3AkcAwUnvH9cAhVTNxrt3Dc94K7J2PsTqwZu++QgihJ6TaHq0gSiI9ZHuGpBOAu4GppIbugcD5koaRSh+n2H5d0onAScAUSQOAp0lJp6hfA2dJegx4DJjc+28SQiiqVaqqahFJpBdsnwKcUsN+7wEHdrD9bODsbj77DKlXV+U4exaPNIRQmhYqZdQikkgIIRQgYODAyCIVkUSaRG6AP6/d5g9sj21EPCGEzkV1VptIIk3C9kPA6EbHEULoRlRnfUQkkRBCKEBESaRaJJEQQiikdQYS1iKSSAghFBSDDdtEEgkhhCJKbhOR9AzwFjALmGl7TLv3BZwMbAe8C+xvu9N58/paJJEQQiigTm0im9ue1sl72wIr5cdY0qDjpum1GdOehBBCQX087cl44FwndwHDJS1R2tF7KZJICCEUNGCAanoAIyXdW/WY0MHhDPxZ0uRO3l8KeL7q9Qt5W1OI6qwWsODIEWzwpX0aHQYA8zTRSN8PZzXPMiuDm+h3eX/G7EaH0HRM1b8VFarOmta+jaMDG9meKmkx4AZJj9u+tWeR9r0oiYQQQgGpTaS86izbU/OfLwOXA+u322UqsEzV66XztqYQSSSEEAqpbUGqWkorkhaQtGDlOWkBu4fb7TYR+LySDYA3bP+77G/VU1GdFUIIBZXYaL44cHlOOIOA39u+TtJBALZPA64hde99ktTF94DSzl6CSCIhhFCEyhtsaPufwFodbD+t6rmBg0s5YR1EEgkhhAJi7qyPiiQSQggFRRJpE0kkhBAKihzSJpJICCEUUWKbSH8QSSSEEApQTAX/EZFEQgihoMghbSKJhBBCQQMii8wRSSSEEAqKHNImkkgIIRQgwcBoWJ8jkkgIIRQUDettIomEEEJBkUPaxCy+BUn6awPO+YykkX193hDCfxK5m28N/7WCli+JSBpke2at+9vesJ7xhBCanBRtIlXmupKIpFGSHq56fbik4yQdKulRSVMkXZTfW0DSmZLulnS/pPF5+/6SJkqaBNzUyXlOkPRAfkyVdFbe/nb+czNJt0r6k6QnJJ0mqdPfU9I2ku6T9KCkm/K2RSRdkWO+S9KaefsISX+W9IikM6DtlkbSPvn7PCDpN5IGdnK+CZUlOT98+/VCv3EIoWt9vMZ6U5vrkkgXvg2sbXtN4KC87Rhgku31gc2Bn+SFXwDWAXazvWlHB7P9Pdujgc2A6cAvOthtfeAQYFVgBWCXjo4laVHgt8CuttcCds9vHQ/cn2M+Gjg3bz8WuN32aqSVzpbNx1kF2AP4VI5tFvC5TuI/3fYY22PmGTq8o11CCD0g0jiRWh6toD9VZ00BLpB0BXBF3rYVME7S4fn1EPIFGbjB9vSuDqjUBeN84Ge2J3ewy915PQAkXQhsBFzawX4bALfafhqg6rwbAbvmbZNyCWQhYBNyQrL9J0mv5f23BNYF7sm9Q+YDXu7qO4QQytci+aEmc2MSmclHS1BD8p/bky6+OwLHSFqDdNOwq+0nqg8gaSzwTg3nOg54wfZZnbzvbl6XTcA5to+q83lCCF2ILr5t5sbqrJeAxfJd+7zADqTvsYztm4EjgWHAUOB64JBcokDS2rWeRNKOwKeBQ7vYbX1Jy+e2kD2A2zvZ7y5gE0nL52MvkrffRq6OkrQZMM32m8CtwN55+7bAwnn/m4DdJC1WOY6k5Wr9TiGE3qsMNqzl0QrmupKI7RmSTgDuBqYCjwMDgfMlDSPdrZ9i+3VJJwInAVPyhf5pUtKpxWHAUsDdOQdNtP29dvvcQ2orWRG4mdR+0VHMr0iaAFyW43gZ+AyppHOmpCmktZP3yx85HrhQ0iPAX4Hn8nEelfQd4M/5ODNIy2Y+W+N3CiGUoDXSQ23muiQCYPsU4JQa9nsPOLCD7WcDZ3fz2c072T606uWbtmtKSravBa5tt206sFMH+75Kas/p6DgXAxfXcs4QQn1EdVabubE6K4QQGib1zqrtUdPxpIF5CMLVHby3v6RXqoYbfKnkr9Nrc2VJpEy5Af68dps/sD22q8/ZvgW4pYPj/Q2Yt93mfW0/1IswQwjNQip7ZcP/Bh4DFurk/Yttf63ME5ap5ZNIvriPLvF4XSafEMLcr6zqLElLk3qW/oDUDjvXieqsEEIooGB11sjKzBH5MaHd4U4CjgBmd3HKXfOsFpdKWqYuX6oXWr4kEkIIRRUoiUyzPaaTY+wAvGx7cu7i35GrgAttfyDpQOAcYIuC4dZVlERCCKEg1fjoxqdIM2o8A1wEbCHp/OodbL9q+4P88gzSjBVNJZJICCEUUNZgQ9tH2V7a9ihgT9I8f/t89FxaourlOFIDfFOJ6qwQQiionuNE8mDqe21PBA6VNI403dN0YP+6nbiHIomEEEJBZeeQ6iED1TNj5HnymnquvEgiIYRQgGidad5rEUmkBSw5bAjf33aVRocBwHzzdLiGVkMMqfecywXMcvMEM3NW88TSlJdqUfZgw7laJJEQQigoeiS1iSQSQggFiJiAsVokkRBCKChqs9pEEgkhhAIq40RCEkkkhBAKihzSJpJICCEUFE0ibSKJhBBCAWkW38giFZFEQgihoOji2yaSSAghFCB1P7liK4kkEkIIBUVtVptIIiGEUFAURNpEEgkhhAKiYf2jIomEEEIRgoHRsj5HJJEQQihIzTm/cENEEgkhhAJSdVajo2geUSjrhKTNJG1Y9fogSZ8v8fijJW3Xi8/vJGnVsuIJIdRugGp7tII+TyKS5pbSz2bAnCRi+zTb55Z4/NFAj5MIsBMQSSSEBpBU06MVdJtEJI2S9HDV68MlHSfpUEmPSpoi6aL83gKSzpR0t6T7JY3P2/eXNFHSJOCmTs4jST+R9LCkhyTtUfXekXnbg5J+lLetKOnGvO0+SSvk0sPVVZ/7haT98/NnJP1vPs7dklbM23eU9Lcc742SFpc0CjgI+IakByRtnL/z4fkzoyXdlb/75ZIWzttvkfTjfPy/S9q4k+86D3ACsEc+/h5d/HYnS/pefr61pFtzCWkc8JP8+RU6OMcESfdKuvf16a9299ccQqiRcsN6LY9W0JtSwbeB5W1/IGl43nYMMMn2F/K2uyXdmN9bB1jT9vROjrcL6e58LWAkcI+kW/O28cBY2+9KWiTvfwHwI9uXSxpCSojLdBPzG7bXyNVSJwE7ALcDG9i2pC8BR9j+pqTTgLdt/xRA0pZVxzkXOMT2XySdABwLfD2/N8j2+rmq6ljg0+2DsP1hTgxjbH8tH/+Hnfx2R+Xf4jbgFGA7209JmghcbfvSjr6o7dOB0wFWWWPt5lnvNIR+oMwuvpIGAvcCU23v0O69eUnXm3WBV4E9bD9T2slL0JtcOQW4QNI+wMy8bSvg25IeAG4BhgDL5vdu6CKBAGwEXGh7lu2XgL8A65EuwmfZfhfA9nRJCwJL2b48b3u/8n43Lqz685P5+dLA9ZIeAr4FrNbVASQNA4bb/kvedA6wSdUul+U/JwOjaoiposPfLn+vLwM3AL+w/VSBY4YQSlZpWC+xTeS/gcc6ee+LwGu2VwR+Dvy411+gZLUkkZnt9huS/9we+CWphHFPbusQsKvt0fmxrO3Kj/NOWUH3INYKd/D8VNLFeQ3gwA4+U9QH+c9ZFCvpdfXbrUG6C1myl7GFEEog1fbo/jhamnQtPaOTXcaTblQBLgW2VJM1ttSSRF4CFpM0IhetdsifW8b2zcCRwDBgKHA9cEjlS0pau0Ast5HaCAZKWpR0d3836Q78AEnz52MuYvst4AVJO+Vt8+b3nwVWza+HA1u2O8ceVX/emZ8PA6bm5/tV7fsWsGD7IG2/AbxW1d6xL6nUVFT743f420laDvgmsDawraSxXcUXQqgvIQaqtgcwstI2mR8T2h3uJOAIYHYnp1sKeB7A9kzgDWBEfb5Zz3R7p2x7Rq73v5t0sX0cGAicn6t2BJxi+3VJJ5J+lCmSBgBPk5JOLS4nVTE9SColHGH7ReA6SaOBeyV9CFwDHE26eP8mxzYD2N32PyX9AXg4n/v+dudYWNIUUmlhr7ztOOASSa8Bk4Dl8/argEtzA/ch7Y6zH3BaTlz/BA6o8TtWu5m26qv/Af7jt5O0I/A74HDb/5L0ReBsSesBFwG/lXQosFtUc4XQR4pVVU2zPabDw0g7AC/bnixps3KC63uyW6PNVdIzpIbsaY2Opa+tssbaPvfKWxodBgCrLNU8hadm+qc/q4mCmTmreWJplnqbLTcZywP3TRbAcqus6WPOuqqmzx34yVGTu0gi/0O6GZ5JqkZfCLjM9j5V+1wPHGf7ztxk8CKwqJvowt0indBCCKEcopw2EdtH2V7a9ihgT1LvzH3a7TaRtmr23fI+TZNAoAHTnkhaAziv3eYPbI/taP+y5L+oPidpa/6zR8XTtnduRDwhhN6r5yy+uYr+XtsTSdXZ50l6EphOSjZNpc+TiO2HSGM/WoLt60mN5iGEfkDAwJJziO1bSF37sf29qu3vA7uXe7ZyzS1TkIQQQnMQLTOlSS0iiYQQQkGRQtpEEgkhhAJiZcOPiiQSQggFtco077WIJBJCCIW0zjTvtYgkEkIIBYgYYFctkkgIIRQUJZE2kURawLyDB7DC4gs0OgwABjXRSj3NNPB31szmiWXGrM7mAux7A5u08aE5o2qMSCIhhFCARGWG3kAkkRBCKCyqs9pEEgkhhIIihbSJJBJCCAVFQaRNJJEQQiggTcAYWaQikkgIIRQiFBVac0QSCSGEgqIg0iaSSAghFJBGrEcWqYgkEkIIRdSw9G0riSQSQggFxVTwbSKJhBBCAWk9kUZH0TwiiYQQQkHRO6tN88yGF0IIcwmptkf3x9EQSXdLelDSI5KO72Cf/SW9IumB/PhSPb5TT0VJJIQQCih5sOEHwBa235Y0GLhd0rW272q338W2v1bWScsUSSSEEAopb7Ch03oEb+eXg/OjedYFqEFUZzUhSbdIGtPoOEIIHaixKisXVkZKurfqMeE/DicNlPQA8DJwg+2/dXDWXSVNkXSppGXq+v0KipJINyQNsj2z0XF0RdJA27MaHUcIraJAOWSa7S5vCPP/u6MlDQcul7S67YerdrkKuND2B5IOBM4BtigcdJ30u5KIpFGSHq56fbik4yQdKunRnM0vyu8tIOnM3LB1v6Txefv+kiZKmgTc1Ml5Bkj6laTHJd0g6RpJu+X31pX0F0mTJV0vaYm8/RZJP87n+7ukjfP2+SRdJOkxSZcD81WdZytJd0q6T9Ilkobm7c/kY90H7N5BfBMqdz+vTptW0q8bQkhdfFXTowjbrwM3A9u02/6q7Q/yyzOAdUv4GqXpd0mkC98G1ra9JnBQ3nYMMMn2+sDmwE8kVdaRXQfYzfamnRxvF2AUsCqwL/BJgNw4dmr+7LrAmcAPqj43KJ/v68CxedtXgHdtr5K3rZuPNRL4DvBp2+sA9wKHVR3rVdvr2L6ofXC2T7c9xvaYESNHdvvjhBBqV2LvrEVzCQRJ8wGfAR5vt88SVS/HAY+V9kVK0ErVWVOACyRdAVyRt20FjJN0eH49BFg2P7/B9vQujrcRcInt2cCLkm7O2z8BrA7ckFc/Gwj8u+pzl+U/J5OSEMAmwCkAtqdImpK3b0BKUnfkY80D3Fl1rIu7/MYhhLoocZzIEsA5kgaSbur/YPtqSScA99qeCBwqaRwwE5gO7F/WycvQH5PITD5awhqS/9yedLHeEThG0hqkkumutp+oPoCkscA7PTy/gEdsf7KT9yvF0ll0//uLlMz26uT9nsYYQuiFsnr42p4CrN3B9u9VPT8KOKqcM5avP1ZnvQQsJmmEpHmBHUjfcxnbNwNHAsOAocD1wCHKt/mS/uMvswt3kHpMDJC0OLBZ3v4EsKikOdVbklbr5li3Anvn/VcH1szb7wI+JWnF/N4Ckj5eIMYQQh2oxkcr6HclEdszclHwbmAqqX5xIHC+pGGkv9tTbL8u6UTgJGCKpAHA06SkU4s/AlsCjwLPA/cBb9j+MDewn5LPNyif45EujvVr4CxJj5HqOyfn7/KKpP2BC3NChNRG8vcaYwwhlEyAYgLGOfpdEgGwfQq5jaGb/d4DDuxg+9nA2d18drakw/NI0xGkpPVQfu8BUtVZ+89sVvV8GrlNJMexZyfnmQSs18H2UV3FF0Kok5gK/iP6ZRLpQ1fnnhXzACfafrHB8YQQ+kDkkDaRRLqRG+DPa7f5A9tjq0sWIYQWEllkjkgi3bD9EDC60XGEEJpF8YGE/VkkkRBCKKCVel7VIpJICCEUFVlkjkgiIYRQUKxs2CaSSAghFBRNIm0iiYQQQhExTuQjIomEEEJBUZ3VJpJICCEUkKY9aXQUzSOSSAt4/8PZPDr1rUaHAcA6o4Y3OoQ5Ppw1u9EhzNFM16R5BjbRvKzN9MNUadKwGiKSSAghFBQTMLaJJBJCCAVFDmkTSSSEEAqKHNImkkgIIRQVWWSOSCIhhFBAmjsrskhFJJEQQihCMCByyBxN1JcvhBDmEiUtsi5piKS7JT0o6RFJx3ewz7ySLpb0pKS/SRpV3hfpvUgiIYRQiGr+rwYfAFvYXou0btE2kjZot88Xgddsrwj8HPhxmd+mtyKJhBBCQVJtj+44eTu/HJwfbrfbeOCc/PxSYEs10UCVSCIhhFBAZdqTGpPISEn3Vj0m/MfxpIGSHgBeBm6w/bd2uywFPA9geybwBjCijl+xkGhYDyGEggr0zppme0xXO9ieBYyWNBy4XNLqth/uZYh9JkoiIYRQUFnVWdVsvw7cDGzT7q2pwDLpvBoEDANe7fWXKEkkkRBCKKikzllIWjSXQJA0H/AZ4PF2u00E9svPdwMm2W7fbtIwUZ0VQghFlLso1RLAOZIGkm7q/2D7akknAPfangj8DjhP0pPAdGDP0s5egkgiIYRQQGpYLyeL2J4CrN3B9u9VPX8f2L2UE9ZBzdVZkkZJ6nVjj6S3u99rzr6bSdqw6vVBkj5fQgzPSBrZ2+OEEFpTWdVZ/UGzl0Q2A94G/gpg+7SGRtNDkgblrnkhhH6geUZpNF7RhvVBki6Q9JikSyXNX31XL2mMpFvy86GSzpL0kKQpknatPpCkkZLulLR9blz6o6R78uNTeWj/QcA3JD0gaWNJx0k6XNKSeVvlMUvSch0dJ59rhKQ/52kFzqCLm4T2Ja58vuPy80MlPZq/z0V52wKSzsxTF9wvaXzevr+kiZImATd1cq6hkm6SdF/+ncZXvfddSU9Iul3ShZIOz9tXkHSdpMmSbpO0cifHnlDpm/76a9O6/EsNIRRT4oj1uV7RksgngC/avkPSmcBXu9j3u8AbttcAkLRw5Q1Ji5N6HHzH9g2Sfg/83PbtkpYFrre9iqTTgLdt/zR/bksA2/8iTRGApIOBTW0/29FxgFWAY4HbbZ8gaXvSNAI98W1gedsfVHpUAMeQekt8IW+7W9KN+b11gDVtT+/keO8DO9t+MyfiuyRNBMYAuwJrkUaw3gdMzp85HTjI9j8kjQV+BWzR/sC2T8/7svLqazdNT44Q+oMoibQpmkSet31Hfn4+cGgX+36aql4Etl/LTweT7swPtv2Xqn1XrWqsWkjS0O6CySWNLwMbdXOcTYBdchx/kvRa+2PVaApwgaQrgCvytq2AcZWSAjAEWDY/v6GLBAKpRPRDSZsAs0kjUxcHPgVcmRvU3pd0Vf6+Q4ENgUuqvuO8PfwuIYQe6MkYkP6saBJpf0drYCZt1WJDajjGTNJd9dZAJYkMADbIF805uuoBIWkJUte3cVVzzxQ+TifxVVfzVX+n7UkJaUfgGElrkBLBrrafaHfOscA73Zzrc8CiwLq2Z0h6hq5/wwHA67ZH1/A9Qgh10ipVVbUo2iayrKRP5ud7A7cDzwDr5m3V7R43AAdXXlRVZxn4ArCypCPztj8Dh1TtOzo/fQtYsH0QkgYDlwBH2v571VudHefWHC+StgUWpnMvAYvldpR5gR3y5wYAy9i+GTiSNGp0KKnK7BDlTCXpP7rrdWEY8HJOIJsDy+XtdwA7Kk0TPbQSg+03gacl7Z7PJUlrFThfCKEM0T1rjqJJ5AngYEmPkS7EvwaOB06WdC8wq2rf7wMLS3pY0oPA5pU38lwxewFbSPoqqVpsTG6wfpTUoA5wFbBzpWG96tgbktoNjq9qXF+yi+McD2wi6RFStdZznX1B2zOAE4C7SYmwMnp0IHC+pIeA+4FT8jQFJ5Kq6Kbk459Yw+9YcUGO9yHg85Vz2b6H1GY0BbgWeIg06Rqk0ssX82/6CGmGzxBCH4oc0kZNNHo+VJE01PbbkuYnlaQm2L6vJ8daefW1fcZlk8oNsIfWGTW80SHM8eGs2Y0OYY5muuDMnNVE14Qm+WG23HgsD9w3WQCj1xnjSbe1n2i3YyOGDprc3QSMc7tmHyfSyk6XtCqpjeScniaQEEK5KlPBh6Rlk4ikEXQ8fmNL26XOkJkb4M9rt/kD22M7+4ztvcuMIYQQ6qFlk0hOFKP76FwP9dW5Qgj1FyWRNi2bREIIoUcEAyKLzBFJJIQQCmilnle1iCQSQghFRRaZI5JICCEUFCPW20QSCSGEgqJJpE0kkRBCKCiSSJtIIiGEUFBUZ7WJaU9agKRXgGd7eZiRQLOsbhWxdKxZYmmWOKC8WJazvSiApOvycWsxzfY2JZy/aUUSCTWRdG+zzAEUsXSsWWJpljiguWLpr4rO4htCCCHMEUkkhBBCj0USCbU6vdEBVIlYOtYssTRLHNBcsfRL0SYSQgihx6IkEkIIocciiYQQQuixSCIhhBB6LJJICCGEHoskEkIIocf+H9+OQErTFl/mAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 288x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = dcn_result[\"model\"][0]\n",
    "mat = model._cross_layer._dense.kernel\n",
    "features = model._all_features\n",
    "\n",
    "block_norm = np.ones([len(features), len(features)])\n",
    "\n",
    "dim = model.embedding_dimension\n",
    "\n",
    "# Compute the norms of the blocks.\n",
    "for i in range(len(features)):\n",
    "  for j in range(len(features)):\n",
    "    block = mat[i * dim:(i + 1) * dim,\n",
    "                j * dim:(j + 1) * dim]\n",
    "    block_norm[i,j] = np.linalg.norm(block, ord=\"fro\")\n",
    "\n",
    "plt.figure(figsize=(9,9))\n",
    "im = plt.matshow(block_norm, cmap=plt.cm.Blues)\n",
    "ax = plt.gca()\n",
    "divider = make_axes_locatable(plt.gca())\n",
    "cax = divider.append_axes(\"right\", size=\"5%\", pad=0.05)\n",
    "plt.colorbar(im, cax=cax)\n",
    "cax.tick_params(labelsize=10) \n",
    "_ = ax.set_xticklabels([\"\"] + features, rotation=45, ha=\"left\", fontsize=10)\n",
    "_ = ax.set_yticklabels([\"\"] + features, fontsize=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pQH-moYd6ZKC"
   },
   "source": [
    "That's all for this notebook! We hope that you have enjoyed learning some basics of DCN and common ways to utilize it. If you are interested in learning more, you could check out two relevant papers: [DCN-v1-paper](https://arxiv.org/pdf/1708.05123.pdf), [DCN-v2-paper](https://arxiv.org/pdf/2008.13535.pdf)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FfAGbq2es2Yn"
   },
   "source": [
    "---\n",
    "\n",
    "\n",
    "##References\n",
    "[DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://arxiv.org/pdf/2008.13535.pdf). \\\n",
    "*Ruoxi Wang, Rakesh Shivanna, Derek Zhiyuan Cheng, Sagar Jain, Dong Lin, Lichan Hong, Ed Chi. (2020)*\n",
    "\n",
    "\n",
    "[Deep & Cross Network for Ad Click Predictions](https://arxiv.org/pdf/1708.05123.pdf). \\\n",
    "*Ruoxi Wang, Bin Fu, Gang Fu, Mingliang Wang. (AdKDD 2017)*"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "dcn.ipynb",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
